00010000:00010000:3.0F:1766096870.908474:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1766096871.933405:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0F:1766096872.768687:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096872.768708:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766096872.768711:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766096872.768722:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1376 00000100:00080000:2.0:1766096872.768727:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:2.0:1766096872.768732:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766096872.768742:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096872.768754:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417120960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096872.768758:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417121088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766096872.768785:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096872.768789:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766096872.768803:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417121024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0F:1766096872.768918:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417121088:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766096872.773293:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417120960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096872.776340:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417121024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096872.788772:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096872.788782:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096872.788792:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417121088:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0F:1766096872.791120:0:2228:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 6/2,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791177:0:2228:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 6/2,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791184:0:2228:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 6/2,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791190:0:2228:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 6/2,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791195:0:2228:0:(ldlm_lock.c:909:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 5/1,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791235:0:2228:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 4/1,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791242:0:2228:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 3/0,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791250:0:2228:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 4/0,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791257:0:2228:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 0000000016a8934f already unused, calling callback (0000000075a89bd0) 00010000:00010000:1.0:1766096872.791261:0:2228:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 5/0,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791275:0:2228:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1766096872.791289:0:2228:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 3/0,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096872.791297:0:2228:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff96a16011a000 lock: 0000000016a8934f/0x2bcbec4ad3fa3c6 lrc: 0/0,0 mode: --/PR res: [0x5a:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19161 timeout: 0 lvb_type: 1 00000100:00100000:1.0:1766096872.791337:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:0:1851885417120832:192.168.203.152@tcp:101:stat.0 00000020:00010000:0.0:1766096872.791944:0:19161:0:(cl_lock.c:53:__cl_lock_trace()) release lock: 000000001064fb65 (00000000b1f09983/0) at cl_lock_release():234 00000020:00010000:0.0:1766096872.791954:0:19161:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: 000000001064fb65 (00000000b1f09983/0) at cl_lock_cancel():151 00000020:00010000:0.0:1766096872.791961:0:19161:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: 00000000fd0dbcd4 (00000000fa4b0a02/1) at cl_lock_cancel():151 00000020:00010000:0.0:1766096872.791966:0:19161:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: 000000001064fb65 (00000000b1f09983/0) at cl_lock_fini():84 00000020:00010000:0.0:1766096872.791969:0:19161:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: 00000000fd0dbcd4 (00000000b1f09983/1) at cl_lock_fini():84 00000080:00200000:0.0:1766096872.792000:0:19161:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x33:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0 00000080:00200000:0.0:1766096872.792007:0:19161:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x33:0x0] ignore/verify layout 0/1, layout version 2 restore needed 0 00000080:00200000:0.0:1766096872.792018:0:19161:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x33:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0 00010000:00010000:1.0:1766096872.959453:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1766096873.980538:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1766096875.004663:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1766096876.029753:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1766096877.054011:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1766096877.885480:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096877.885487:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766096877.885491:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766096877.885502:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1381 00000100:00080000:1.0:1766096877.885506:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1766096877.885511:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766096877.885521:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766096877.885536:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417121280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766096877.885555:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096877.885557:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766096877.885578:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417121216:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766096877.885659:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766096877.885666:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417121152:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766096877.888644:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417121280:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766096877.891793:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096877.891803:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096877.891812:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417121280:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766096877.893546:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417121216:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766096877.897566:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417121152:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766096878.077504:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000080:0.0:1766096878.185935:0:19849:0:(module.c:221:libcfs_ioctl()) libcfs ioctl cmd 3221775648 00000001:02000400:0.0:1766096878.185941:0:19849:0:(debug.c:702:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:27:57 (1766096877) 00000080:00200000:0.0:1766096878.378186:0:19861:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:0.0:1766096878.378209:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1766096878.378224:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.378236:0:19861:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1766096878.378245:0:19861:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:0.0:1766096878.378247:0:19861:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:0.0:1766096878.378254:0:19861:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 0000000073b43a59 00010000:00010000:0.0:1766096878.378256:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.378268:0:19861:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1766096878.378283:0:19861:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:0.0:1766096878.378294:0:19861:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), flags=8 00000080:00200000:0.0:1766096878.378299:0:19861:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=getattr 00000002:00010000:0.0:1766096878.378309:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1766096878.378371:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.378378:0:19861:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.378388:0:19861:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1766096878.378412:0:19861:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121344:192.168.203.152@tcp:101:rm.0 00000100:00100000:0.0:1766096878.378485:0:19861:0:(client.c:2500:ptlrpc_set_wait()) set 000000000c37d539 going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.394757:0:19861:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121344:192.168.203.152@tcp:101:rm.0 00010000:00010000:0.0:1766096878.394779:0:19861:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394794:0:19861:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394800:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394822:0:19861:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394829:0:19861:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394836:0:19861:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 000000001bc2e436 already unused, calling callback (00000000d04820b2) 00010000:00010000:0.0:1766096878.394840:0:19861:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394845:0:19861:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394852:0:19861:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394859:0:19861:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1766096878.394863:0:19861:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.394870:0:19861:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3cd lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.394898:0:19861:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@000000007e872964 x1851885417121344/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 584/640 e 0 to 0 dl 1766097268 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:0.0:1766096878.394926:0:19861:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:0.0:1766096878.395003:0:19861:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:0.0:1766096878.395015:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1766096878.395027:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.395035:0:19861:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1766096878.395043:0:19861:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:0.0:1766096878.395045:0:19861:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:0.0:1766096878.395050:0:19861:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 00000000901ba391 00010000:00010000:0.0:1766096878.395052:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.395057:0:19861:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1766096878.395068:0:19861:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:0.0:1766096878.395082:0:19861:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), flags=0 00000080:00200000:0.0:1766096878.395087:0:19861:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=getattr 00000002:00010000:0.0:1766096878.395107:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1766096878.395138:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.395144:0:19861:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.395153:0:19861:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1766096878.395170:0:19861:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121408:192.168.203.152@tcp:101:rm.0 00000100:00100000:0.0:1766096878.395238:0:19861:0:(client.c:2500:ptlrpc_set_wait()) set 000000000c37d539 going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.401579:0:19861:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121408:192.168.203.152@tcp:101:rm.0 00010000:00010000:0.0:1766096878.401599:0:19861:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401613:0:19861:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401619:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401626:0:19861:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401634:0:19861:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401640:0:19861:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000998a4c47 already unused, calling callback (00000000d04820b2) 00010000:00010000:0.0:1766096878.401644:0:19861:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401650:0:19861:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401656:0:19861:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401669:0:19861:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1766096878.401673:0:19861:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401693:0:19861:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000998a4c47/0x2bcbec4ad3fa3d4 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.401709:0:19861:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@000000007e872964 x1851885417121408/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 584/640 e 0 to 0 dl 1766097268 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:0.0:1766096878.401735:0:19861:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:0.0:1766096878.401795:0:19861:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:0.0:1766096878.401806:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1766096878.401812:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401821:0:19861:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1766096878.401828:0:19861:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:0.0:1766096878.401831:0:19861:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:0.0:1766096878.401836:0:19861:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 0000000073b43a59 00010000:00010000:0.0:1766096878.401838:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401844:0:19861:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1766096878.401855:0:19861:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:0.0:1766096878.401868:0:19861:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000ec967eae), flags=8 00000080:00200000:0.0:1766096878.401885:0:19861:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=getattr 00000002:00010000:0.0:1766096878.401896:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1766096878.401936:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401942:0:19861:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.401961:0:19861:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1766096878.401979:0:19861:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121472:192.168.203.152@tcp:101:rm.0 00000100:00100000:0.0:1766096878.402049:0:19861:0:(client.c:2500:ptlrpc_set_wait()) set 000000000c37d539 going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.405894:0:19861:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121472:192.168.203.152@tcp:101:rm.0 00010000:00010000:0.0:1766096878.405914:0:19861:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405926:0:19861:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405932:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405937:0:19861:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405944:0:19861:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405957:0:19861:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000a7785544 already unused, calling callback (00000000d04820b2) 00010000:00010000:0.0:1766096878.405960:0:19861:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405979:0:19861:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405985:0:19861:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.405991:0:19861:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1766096878.405996:0:19861:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.406001:0:19861:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000a7785544/0x2bcbec4ad3fa3db lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.406015:0:19861:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@000000007e872964 x1851885417121472/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 592/640 e 0 to 0 dl 1766097268 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:0.0:1766096878.406043:0:19861:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:0.0:1766096878.406103:0:19861:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:0.0:1766096878.406112:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1766096878.406117:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.406124:0:19861:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1766096878.406132:0:19861:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:0.0:1766096878.406134:0:19861:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:0.0:1766096878.406138:0:19861:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 00000000901ba391 00010000:00010000:0.0:1766096878.406140:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.406151:0:19861:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1766096878.406162:0:19861:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:0.0:1766096878.406177:0:19861:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000ec967eae), flags=0 00000080:00200000:0.0:1766096878.406180:0:19861:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=getattr 00000002:00010000:0.0:1766096878.406189:0:19861:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1766096878.406216:0:19861:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.406222:0:19861:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.406229:0:19861:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1766096878.406244:0:19861:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121536:192.168.203.152@tcp:101:rm.0 00000100:00100000:0.0:1766096878.406301:0:19861:0:(client.c:2500:ptlrpc_set_wait()) set 000000000c37d539 going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.416139:0:19861:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job rm:56772462-1282-4783-8e1a-69be7ca0dd12:19861:1851885417121536:192.168.203.152@tcp:101:rm.0 00010000:00010000:0.0:1766096878.416159:0:19861:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416173:0:19861:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416179:0:19861:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416202:0:19861:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416210:0:19861:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416216:0:19861:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 000000001bc2e436 already unused, calling callback (00000000d04820b2) 00010000:00010000:0.0:1766096878.416219:0:19861:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416225:0:19861:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416232:0:19861:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416238:0:19861:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1766096878.416244:0:19861:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.416250:0:19861:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3e2 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19861 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.416266:0:19861:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@000000007e872964 x1851885417121536/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 592/640 e 0 to 0 dl 1766097268 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:0.0:1766096878.416293:0:19861:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:2.0:1766096878.442139:0:19864:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:2.0:1766096878.442164:0:19864:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1766096878.442180:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1766096878.442193:0:19864:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:2.0:1766096878.442203:0:19864:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:2.0:1766096878.442206:0:19864:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:2.0:1766096878.442213:0:19864:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 00000000b1a90812 00010000:00010000:2.0:1766096878.442215:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1766096878.442230:0:19864:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1766096878.442245:0:19864:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:2.0:1766096878.442257:0:19864:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), flags=8 00000080:00200000:2.0:1766096878.442262:0:19864:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=getattr 00000002:00010000:2.0:1766096878.442273:0:19864:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1766096878.442314:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1766096878.442321:0:19864:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1766096878.442330:0:19864:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1766096878.442396:0:19864:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417121600:192.168.203.152@tcp:101:lfs.0 00000100:00100000:2.0:1766096878.442477:0:19864:0:(client.c:2500:ptlrpc_set_wait()) set 00000000cf2159bd going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.449216:0:19864:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417121600:192.168.203.152@tcp:101:lfs.0 00010000:00010000:0.0:1766096878.449236:0:19864:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449262:0:19864:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449269:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449275:0:19864:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449283:0:19864:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449288:0:19864:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000c8464653 already unused, calling callback (00000000d04820b2) 00010000:00010000:0.0:1766096878.449292:0:19864:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449298:0:19864:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449305:0:19864:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449313:0:19864:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1766096878.449318:0:19864:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.449324:0:19864:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000c8464653/0x2bcbec4ad3fa3e9 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.450478:0:19864:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@00000000e377dbf8 x1851885417121600/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 584/640 e 0 to 0 dl 1766097268 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'lfs.0' 00010000:00010000:0.0:1766096878.450520:0:19864:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:1.0:1766096878.451920:0:19864:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:1.0:1766096878.451943:0:19864:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1766096878.451964:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.451977:0:19864:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:1.0:1766096878.451986:0:19864:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:1.0:1766096878.451990:0:19864:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:1.0:1766096878.451996:0:19864:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 000000007fc77cfe 00010000:00010000:1.0:1766096878.451998:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.452005:0:19864:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1766096878.452019:0:19864:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:1.0:1766096878.452034:0:19864:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:1.0:1766096878.452039:0:19864:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1766096878.452044:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.452050:0:19864:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:1.0:1766096878.452072:0:19864:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:1.0:1766096878.452074:0:19864:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:1.0:1766096878.452196:0:19864:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 000000003464b78e 00010000:00010000:1.0:1766096878.452199:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.452205:0:19864:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1766096878.452212:0:19864:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 3 00000080:00200000:1.0:1766096878.452216:0:19864:0:(namei.c:1183:ll_atomic_open()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), file 0000000024d628b6, open_flags a141, mode 81b6 opened 0 00000080:00200000:1.0:1766096878.452225:0:19864:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=open|creat 00000002:00010000:1.0:1766096878.452239:0:19864:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x200000401:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:1.0:1766096878.452280:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.452287:0:19864:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000020:00100000:1.0:1766096878.452297:0:19864:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff96a16011a000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1766096878.452306:0:19864:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@00000000174f4196 x1851885417121728/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 528/67832 e 0 to 0 dl 0 ref 1 fl New:PQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1766096878.452778:0:19864:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1766096878.452800:0:19864:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417121728:192.168.203.152@tcp:101:lfs.0 00000100:00100000:1.0:1766096878.452944:0:19864:0:(client.c:2500:ptlrpc_set_wait()) set 00000000a501aee3 going to sleep for 388 seconds 00000100:00100000:1.0:1766096878.458716:0:19864:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417121728:192.168.203.152@tcp:101:lfs.0 00010000:00010000:1.0:1766096878.458741:0:19864:0:(ldlm_request.c:714:ldlm_cli_enqueue_fini()) ### server returned different mode CR ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458759:0:19864:0:(ldlm_request.c:729:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458766:0:19864:0:(ldlm_request.c:742:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458776:0:19864:0:(ldlm_lock.c:1064:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458785:0:19864:0:(ldlm_request.c:796:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458795:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458801:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458807:0:19864:0:(ldlm_lock.c:909:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1766096878.458816:0:19864:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=3 disposition=8000037, status=0 req@00000000174f4196 x1851885417121728/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 528/600 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'lfs.0' 00000002:00100000:1.0:1766096878.458826:0:19864:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000174f4196 x1851885417121728/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 528/600 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'lfs.0' 00000002:00010000:1.0:1766096878.458849:0:19864:0:(mdc_locks.c:1179:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.458859:0:19864:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 16 res 8589935617/52 (0 0) 00000080:00200000:1.0:1766096878.458882:0:19864:0:(namei.c:1412:ll_create_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=open|creat 00000080:00200000:1.0:1766096878.458908:0:19864:0:(llite_lib.c:2838:ll_read_inode2()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89) 00000080:00200000:1.0:1766096878.458914:0:19864:0:(llite_lib.c:2760:ll_update_inode()) inode=[0x200000401:0x34:0x0], updating i_size 0 00000080:00200000:1.0:1766096878.458955:0:19864:0:(file.c:5878:ll_layout_conf()) [0x200000401:0x34:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1766096878.458961:0:19864:0:(namei.c:1379:ll_create_node()) setting l_ast_data to inode [0x200000401:0x34:0x0](000000007147ad89) 00000080:00010000:1.0:1766096878.458964:0:19864:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000401:0x34:0x0](000000007147ad89) for lock 0x2bcbec4ad3fa3f0 00000080:00200000:1.0:1766096878.458978:0:19864:0:(file.c:837:ll_file_open()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89), flags 120501 00000002:00100000:1.0:1766096878.458993:0:19864:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000174f4196 x1851885417121728/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 528/600 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'lfs.0' 00000080:00010000:1.0:1766096878.459056:0:19864:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa3f0 from it 0000000087b0dade 00010000:00010000:1.0:1766096878.459059:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.459066:0:19864:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1766096878.459081:0:19864:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89), inode mode 81a4 mask 40 00000080:00200000:1.0:1766096878.459122:0:19864:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89), inode mode 81a4 mask 2 00000080:00200000:1.0:1766096878.459131:0:19864:0:(xattr.c:358:ll_xattr_set()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89), xattr lov 00000002:00010000:1.0:1766096878.459144:0:19864:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000401:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:1.0:1766096878.459186:0:19864:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.459203:0:19864:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000020:00100000:1.0:1766096878.459213:0:19864:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff96a16011a000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1766096878.459224:0:19864:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@00000000baf3fdb7 x1851885417121856/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 2216/133368 e 0 to 0 dl 0 ref 1 fl New:PQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1766096878.459231:0:19864:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1766096878.459248:0:19864:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417121856:192.168.203.152@tcp:101:lfs.0 00000100:00100000:1.0:1766096878.459427:0:19864:0:(client.c:2500:ptlrpc_set_wait()) set 00000000a501aee3 going to sleep for 388 seconds 00000100:00100000:2.0:1766096878.463740:0:2222:0:(events.c:361:request_in_callback()) peer: 12345-192.168.203.152@tcp (source: 12345-192.168.203.152@tcp) 00000100:00100000:3.0:1766096878.463822:0:5004:0:(service.c:2152:ptlrpc_server_handle_req_in()) got req x1851885432767808 00000100:00100000:3.0:1766096878.463882:0:5004:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.152@tcp, seq: 322 00000100:00100000:3.0:1766096878.463889:0:5004:0:(service.c:2324:ptlrpc_server_handle_request()) Handling RPC req@00000000b60af682 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:56772462-1282-4783-8e1a-69be7ca0dd12+5:5513:x1851885432767808:12345-192.168.203.152@tcp:104: 00010000:00010000:3.0:1766096878.463905:0:5004:0:(ldlm_lockd.c:2468:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1766096878.464008:0:5004:0:(ldlm_lockd.c:2159:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1766096878.464014:0:5004:0:(service.c:2372:ptlrpc_server_handle_request()) Handled RPC req@00000000b60af682 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:56772462-1282-4783-8e1a-69be7ca0dd12+5:5513:x1851885432767808:12345-192.168.203.152@tcp:104: Request processed in 124us (282us total) trans 0 rc 0/0 00000100:00100000:3.0:1766096878.464020:0:5004:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.152@tcp, seq: 322 00010000:00010000:1.0:1766096878.464209:0:5013:0:(ldlm_lockd.c:2789:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1766096878.464218:0:5013:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.464227:0:5013:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000942f4f14 already unused, calling callback (00000000d04820b2) 00010000:00010000:1.0:1766096878.464231:0:5013:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.464238:0:5013:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.464245:0:5013:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 3/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1766096878.464276:0:5013:0:(vvp_object.c:137:vvp_conf_set()) [0x200000401:0x34:0x0]: losing layout lock 00010000:00010000:1.0:1766096878.464314:0:5013:0:(ldlm_request.c:1279:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 2/0,0 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.464338:0:5013:0:(ldlm_request.c:1325:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1766096878.464411:0:5013:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 1/0,0 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.464418:0:5013:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa3f0 lrc: 0/0,0 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x1fc4c92446a1ea72 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1766096878.464435:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1766096878.464437:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1766096878.464438:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096878.464445:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766096878.464461:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035c3adf0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417121920:192.168.203.152@tcp:103:ldlm_bl_02.0 00000100:00100000:0.0:1766096878.470903:0:19864:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417121856:192.168.203.152@tcp:101:lfs.0 00010000:00010000:0.0:1766096878.470927:0:19864:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470939:0:19864:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470955:0:19864:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470962:0:19864:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470969:0:19864:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470974:0:19864:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 0000000056142f36 already unused, calling callback (00000000d04820b2) 00010000:00010000:0.0:1766096878.470978:0:19864:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470984:0:19864:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.470990:0:19864:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.471011:0:19864:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1766096878.471016:0:19864:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.471021:0:19864:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 0000000056142f36/0x2bcbec4ad3fa3f7 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19864 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.471036:0:19864:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=1 disposition=800002b, status=0 req@00000000baf3fdb7 x1851885417121856/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 2216/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000002:00100000:0.0:1766096878.471047:0:19864:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000baf3fdb7 x1851885417121856/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 2216/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000080:00200000:0.0:1766096878.471094:0:19864:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1766096878.471102:0:19864:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0 00000080:00200000:0.0:1766096878.471137:0:19864:0:(llite_lib.c:2760:ll_update_inode()) inode=[0x200000401:0x34:0x0], updating i_size 0 00000002:00100000:0.0:1766096878.471149:0:19864:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000baf3fdb7 x1851885417121856/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 656/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000002:00100000:0.0:1766096878.471167:0:19864:0:(mdc_request.c:944:mdc_close()) @@@ matched open req@00000000baf3fdb7 x1851885417121856/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 656/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000020:00100000:0.0:1766096878.471187:0:19864:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff96a16011a000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1766096878.471191:0:19864:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000007e872964 x1851885417122048/t0(0) o35->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766096878.471208:0:19864:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417122048:192.168.203.152@tcp:35:lfs.0 00000100:00100000:1.0:1766096878.471772:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035c3adf0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417121920:192.168.203.152@tcp:103:ldlm_bl_02.0 00000100:00100000:0.0:1766096878.471847:0:19864:0:(client.c:2500:ptlrpc_set_wait()) set 000000007be5163e going to sleep for 6 seconds 00000100:00100000:0.0:1766096878.475124:0:19864:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417122048:192.168.203.152@tcp:35:lfs.0 00000002:00100000:0.0:1766096878.475153:0:19864:0:(mdc_request.c:854:mdc_free_open()) @@@ free open request, rq_replay=0 req@00000000baf3fdb7 x1851885417121856/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 656/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RQU/4/0 rc 301/301 job:'lfs.0' 00000080:00200000:0.0:1766096878.475214:0:19864:0:(file.c:410:ll_file_release()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89) 00000002:00100000:0.0:1766096878.475245:0:19864:0:(mdc_request.c:944:mdc_close()) @@@ matched open req@00000000174f4196 x1851885417121728/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 528/600 e 0 to 0 dl 1766097268 ref 1 fl Complete:RPQU/4/ffffffff rc 0/-1 job:'lfs.0' 00000020:00100000:0.0:1766096878.475267:0:19864:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff96a16011a000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1766096878.475271:0:19864:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@00000000d2c84c91 x1851885417122176/t0(0) o35->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766096878.475287:0:19864:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417122176:192.168.203.152@tcp:35:lfs.0 00000100:00100000:0.0:1766096878.475349:0:19864:0:(client.c:2500:ptlrpc_set_wait()) set 000000007be5163e going to sleep for 6 seconds 00000100:00100000:0.0:1766096878.478013:0:19864:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job lfs:56772462-1282-4783-8e1a-69be7ca0dd12:19864:1851885417122176:192.168.203.152@tcp:35:lfs.0 00000002:00100000:0.0:1766096878.478043:0:19864:0:(mdc_request.c:854:mdc_free_open()) @@@ free open request, rq_replay=0 req@00000000174f4196 x1851885417121728/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 528/600 e 0 to 0 dl 1766097268 ref 1 fl Complete:RQU/4/ffffffff rc 0/-1 job:'lfs.0' 00000080:00200000:0.0:1766096878.488511:0:19865:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:0.0:1766096878.488535:0:19865:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1766096878.488549:0:19865:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.488564:0:19865:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1766096878.488573:0:19865:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:0.0:1766096878.488576:0:19865:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:0.0:1766096878.488582:0:19865:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 0000000066388e65 00010000:00010000:0.0:1766096878.488584:0:19865:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.488599:0:19865:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1766096878.488623:0:19865:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 1 00000080:00200000:0.0:1766096878.488635:0:19865:0:(file.c:5235:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae),name=/ 00000002:00010000:0.0:1766096878.488640:0:19865:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1766096878.488646:0:19865:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.488652:0:19865:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1766096878.488658:0:19865:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) 00000080:00010000:0.0:1766096878.488660:0:19865:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000ec967eae) for lock 0x2bcbec4ad3fa38e 00000080:00010000:0.0:1766096878.488663:0:19865:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa38e from it 00000000f00edf21 00010000:00010000:0.0:1766096878.488664:0:19865:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.488669:0:19865:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000009829d258/0x2bcbec4ad3fa38e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1e93e expref: -99 pid: 18078 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1766096878.488676:0:19865:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000ec967eae), inode mode 41ed mask 3 00000080:00200000:0.0:1766096878.488680:0:19865:0:(namei.c:1183:ll_atomic_open()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), file 000000000530e29f, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1766096878.488689:0:19865:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000ec967eae), intent=open|creat 00000002:00010000:0.0:1766096878.488704:0:19865:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x200000401:0x35:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1766096878.488749:0:19865:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.488755:0:19865:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1766096878.488777:0:19865:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff96a16011a000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1766096878.488786:0:19865:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@00000000e978fab0 x1851885417122304/t0(0) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 2224/67832 e 0 to 0 dl 0 ref 1 fl New:PQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:0.0:1766096878.488794:0:19865:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1766096878.488814:0:19865:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e978fab0 pname:cluuid:pid:xid:nid:opc:job dd:56772462-1282-4783-8e1a-69be7ca0dd12:19865:1851885417122304:192.168.203.152@tcp:101:dd.0 00000100:00100000:0.0:1766096878.489044:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 000000007be5163e going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.493960:0:19865:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e978fab0 pname:cluuid:pid:xid:nid:opc:job dd:56772462-1282-4783-8e1a-69be7ca0dd12:19865:1851885417122304:192.168.203.152@tcp:101:dd.0 00010000:00010000:0.0:1766096878.493985:0:19865:0:(ldlm_request.c:714:ldlm_cli_enqueue_fini()) ### server returned different mode CR ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494005:0:19865:0:(ldlm_request.c:729:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494011:0:19865:0:(ldlm_request.c:742:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494020:0:19865:0:(ldlm_lock.c:1064:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494029:0:19865:0:(ldlm_request.c:796:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494040:0:19865:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494064:0:19865:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1766096878.494071:0:19865:0:(ldlm_lock.c:909:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1766096878.494080:0:19865:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=3 disposition=2b, status=0 req@00000000e978fab0 x1851885417122304/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 2224/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'dd.0' 00000002:00100000:0.0:1766096878.494091:0:19865:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000e978fab0 x1851885417122304/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 2224/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'dd.0' 00000002:00010000:0.0:1766096878.494103:0:19865:0:(mdc_locks.c:847:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1766096878.494113:0:19865:0:(mdc_locks.c:1179:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1766096878.494121:0:19865:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 000000001e9eca16 type 13 mode 16 res 8589935617/52 (0 0) 00000080:00200000:0.0:1766096878.494161:0:19865:0:(namei.c:157:ll_iget()) got inode: [0x200000401:0x34:0x0](000000007147ad89): rc = 0 00000080:00200000:0.0:1766096878.494172:0:19865:0:(file.c:5878:ll_layout_conf()) [0x200000401:0x34:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:0.0:1766096878.494177:0:19865:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000401:0x34:0x0](000000007147ad89) for lock 0x2bcbec4ad3fa3fe 00000080:00010000:0.0:1766096878.494186:0:19865:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x34:0x0](000000007147ad89) 00000080:00200000:0.0:1766096878.494200:0:19865:0:(file.c:837:ll_file_open()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89), flags 101101 00000002:00100000:0.0:1766096878.494213:0:19865:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000e978fab0 x1851885417122304/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 664/736 e 0 to 0 dl 1766097268 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'dd.0' 00000080:00010000:0.0:1766096878.494272:0:19865:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x2bcbec4ad3fa3fe from it 00000000eb29156a 00010000:00010000:0.0:1766096878.494275:0:19865:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1766096878.494286:0:19865:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff96a16011a000 lock: 000000001bc2e436/0x2bcbec4ad3fa3fe lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x1fc4c92446a1eaaa expref: -99 pid: 19865 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1766096878.494301:0:19865:0:(file.c:5669:ll_inode_permission()) VFS Op:inode=[0x200000401:0x34:0x0](000000007147ad89), inode mode 81a4 mask 42 00000080:00200000:0.0:1766096878.494312:0:19865:0:(llite_lib.c:2191:ll_setattr_raw()) lustre: setattr inode [0x200000401:0x34:0x0](000000007147ad89) from 0 to 0, valid a068, hsm_import 0 00000020:00100000:0.0:1766096878.494349:0:19865:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff96a16011a000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1766096878.494512:0:19865:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000008ffe0f9f x1851885417122432/t0(0) o36->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766096878.494533:0:19865:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job dd:56772462-1282-4783-8e1a-69be7ca0dd12:19865:1851885417122432:192.168.203.152@tcp:36:dd.0 00000100:00100000:0.0:1766096878.494766:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 0000000029e3ea55 going to sleep for 388 seconds 00000100:00100000:0.0:1766096878.509412:0:19865:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job dd:56772462-1282-4783-8e1a-69be7ca0dd12:19865:1851885417122432:192.168.203.152@tcp:36:dd.0 00000080:00200000:1.0:1766096878.509502:0:19865:0:(file.c:1543:ll_io_set_mirror()) f32.sanity-flr: desiginated mirror: 0 00000080:00200000:1.0:1766096878.509508:0:19865:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x34:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:1.0:1766096878.509524:0:19865:0:(lov_io.c:838:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1766096878.509537:0:19865:0:(vvp_io.c:222:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:1.0:1766096878.509549:0:19865:0:(lustre_osc.h:181:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:1.0:1766096878.509552:0:19865:0:(osc_lock.c:1241:osc_lock_init()) ### lock 00000000262f3008, osc lock 000000006c751544, flags 0x80000000 00010000:00010000:1.0:1766096878.509578:0:19865:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa405 lrc: 3/0,1 mode: --/PW res: [0x5b:0x0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19865 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096878.509588:0:19865:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa405 lrc: 3/0,1 mode: --/PW res: [0x5b:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19865 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1766096878.509615:0:19865:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff96a16011a000 lock: 00000000942f4f14/0x2bcbec4ad3fa405 lrc: 3/0,1 mode: --/PW res: [0x5b:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19865 timeout: 0 lvb_type: 1 00000100:00080000:1.0:1766096878.509648:0:19865:0:(client.c:1659:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@0000000035c3adf0 x1851885417122496/t0(0) o101->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/0/ffffffff rc 0/-1 job:'dd.0' 00000100:00100000:1.0:1766096878.509657:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096879.100533:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096879.548652:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096880.125384:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096880.573022:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096881.150401:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096881.599130:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096882.172571:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096882.620468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096883.006494:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096883.006501:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096883.006504:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096883.006514:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1386 00000100:00080000:0.0:1766096883.006519:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096883.006524:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096883.006534:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096883.006550:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417122688/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766096883.006569:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096883.006573:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096883.006600:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417122624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766096883.006715:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766096883.006721:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417122560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766096883.008230:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417122688:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766096883.009551:0:2228:0:(client.c:2868:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff96a16011a000: committing for last_committed 4294967807 gen 1 00000100:00100000:0.0:1766096883.009678:0:2228:0:(client.c:2899:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@00000000e978fab0 x1851885417122304/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff96a16011a000@192.168.203.152@tcp:12/10 lens 664/736 e 0 to 0 dl 1766097268 ref 1 fl Complete:RPQU/4/ffffffff rc 0/-1 job:'dd.0' 00000100:00100000:0.0:1766096883.009693:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417122624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766096883.010596:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417122560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766096883.042923:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096883.042933:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096883.042955:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417122688:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766096883.197154:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096883.644607:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096884.221890:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096884.671705:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096885.244582:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096885.692484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096886.270676:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096886.716469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096887.293496:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096887.740473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096888.130985:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096888.130991:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096888.130995:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096888.131004:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1392 00000100:00080000:0.0:1766096888.131007:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096888.131013:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096888.131023:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096888.131037:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ffe0f9f x1851885417122880/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766096888.131052:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417122752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096888.131059:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096888.131063:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096888.131084:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417122816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766096888.131220:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417122880:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766096888.133656:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096888.133664:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096888.133671:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417122880:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766096888.137573:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417122752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766096888.139594:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417122816:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766096888.318089:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096888.764539:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096889.343214:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096889.788455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096890.367336:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096890.813866:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096891.388559:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096891.836462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096892.412593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096892.860477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766096893.245299:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096893.245307:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096893.245311:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096893.245328:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1397 00000100:00080000:3.0:1766096893.245332:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096893.245337:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096893.245347:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096893.245396:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417123072/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766096893.245412:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096893.245415:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096893.245470:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417122944:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766096893.245586:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766096893.245592:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417123008:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766096893.246650:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123072:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766096893.254255:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417123008:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766096893.254277:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417122944:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766096893.255335:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096893.255346:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096893.255382:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123072:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766096893.437829:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096893.886021:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096894.460602:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096894.908662:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096895.484542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096895.932471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096896.509281:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096896.956475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096897.532584:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096897.980474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096898.364563:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096898.364571:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096898.364575:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096898.364585:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1402 00000100:00080000:0.0:1766096898.364589:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096898.364630:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096898.364640:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096898.364658:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417123264/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766096898.364674:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096898.364677:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096898.364698:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417123200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096898.364849:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766096898.364856:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417123136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096898.364892:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123264:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096898.372059:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417123136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096898.372081:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417123200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096898.372165:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766096898.372170:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766096898.372174:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123264:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766096898.556837:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096899.006259:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096899.582976:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096900.031859:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096900.605117:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096901.052583:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096901.629021:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096902.079483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096902.652569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096903.100678:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766096903.485951:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096903.485957:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096903.485961:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096903.485970:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1407 00000100:00080000:3.0:1766096903.485974:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096903.485979:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096903.485989:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096903.486003:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417123456/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766096903.486028:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096903.486031:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766096903.489088:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417123328:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766096903.489248:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766096903.489256:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417123392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766096903.491121:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123456:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766096903.498748:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417123392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766096903.499509:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766096903.499516:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766096903.499521:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123456:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766096903.502409:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417123328:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766096903.679922:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096904.124472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096904.701127:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096905.148471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096905.724795:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096906.172887:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096906.748554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096907.196467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096907.772511:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096908.222313:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096908.604637:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096908.604643:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096908.604647:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096908.604657:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1412 00000100:00080000:0.0:1766096908.604660:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096908.604665:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096908.604674:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096908.604691:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d602f5b x1851885417123648/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766096908.604693:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417123520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766096908.604722:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096908.604736:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096908.604751:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417123584:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766096908.604897:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123648:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096908.613923:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417123584:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096908.617565:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417123520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096908.618024:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096908.618030:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096908.618053:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123648:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766096908.796693:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1766096909.116589:0:15096:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00020000:00200000:0.0:1766096909.116609:0:15096:0:(lov_io.c:838:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1766096909.116636:0:15096:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0 00000100:00100000:2.0:1766096909.244953:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096909.821710:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096910.268545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096910.848670:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096911.292471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096911.868599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096912.316617:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096912.893726:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096913.344343:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766096913.724623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096913.724640:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766096913.724643:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766096913.724657:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1417 00000100:00080000:2.0:1766096913.724661:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766096913.724666:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766096913.724675:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766096913.724720:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417123840/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766096913.724745:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096913.724749:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766096913.724769:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417123776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766096913.724846:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766096913.724852:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417123712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096913.724971:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123840:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766096913.731083:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417123712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766096913.732454:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417123776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766096913.732571:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096913.732575:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096913.732579:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417123840:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766096913.916497:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096914.364526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096914.941560:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096915.391976:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096915.964816:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096916.412465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096916.991157:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096917.436507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096918.012597:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096918.460441:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766096918.844538:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096918.844545:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766096918.844549:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766096918.844558:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1422 00000100:00080000:1.0:1766096918.844561:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766096918.844567:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766096918.844576:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766096918.844590:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417124032/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766096918.844613:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096918.844630:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096918.844692:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124032:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096918.844740:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417123904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096918.844799:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766096918.844808:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417123968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096918.848301:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417123968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096918.848318:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417123904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096918.849436:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096918.849442:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096918.849448:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124032:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766096919.041914:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096919.484555:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096920.065514:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096920.508549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096921.084530:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096921.532462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096922.108536:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096922.556470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096923.134597:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096923.580463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096923.964563:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096923.964570:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096923.964574:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096923.964583:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1427 00000100:00080000:0.0:1766096923.964586:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096923.964590:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096923.964601:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096923.964623:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417124224/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766096923.964698:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124224:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766096923.966181:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417124096:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766096923.966252:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766096923.966259:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417124160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096923.967859:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096923.967866:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:3.0:1766096923.975584:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096923.975595:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096923.975604:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124224:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766096923.976704:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417124096:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766096923.977570:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417124160:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766096924.156601:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096924.604558:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096925.181812:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096925.632923:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096926.204714:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096926.652519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096927.229201:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096927.676543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096928.260926:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096928.700647:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766096929.084705:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096929.084714:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766096929.084719:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766096929.084733:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1433 00000100:00080000:1.0:1766096929.084737:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766096929.084744:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766096929.084759:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766096929.084781:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417124416/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766096929.084867:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124416:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096929.085787:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417124288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096929.086289:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096929.086313:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766096929.086909:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417124352:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096929.096128:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096929.096155:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096929.096273:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417124288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096929.096396:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417124352:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766096929.096527:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124416:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766096929.278128:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096929.724649:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096930.300623:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096930.748641:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096931.324552:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096931.772535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096932.348870:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096932.797054:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096933.375685:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096933.820463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766096934.204601:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417124480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096934.205821:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096934.205828:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096934.205831:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096934.205839:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1438 00000100:00080000:3.0:1766096934.205842:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096934.205849:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096934.205860:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096934.205876:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417124608/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766096934.205927:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124608:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766096934.206018:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417124544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096934.206382:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096934.206386:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096934.212957:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417124480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766096934.219173:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417124544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766096934.251787:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096934.251796:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096934.251805:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124608:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766096934.397006:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096934.845165:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096935.422688:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096935.870789:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096936.445694:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096936.892470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096937.472780:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096937.916419:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096938.494620:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096938.940597:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766096939.324574:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096939.324580:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096939.324583:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096939.324594:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1443 00000100:00080000:3.0:1766096939.324609:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096939.324614:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096939.324624:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096939.324637:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417124800/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766096939.324660:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096939.324663:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096939.324686:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417124736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766096939.324760:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766096939.324764:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417124672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096939.325301:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124800:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766096939.331241:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417124736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766096939.333909:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417124672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766096939.335989:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096939.336000:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096939.336009:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124800:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766096939.519953:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096939.965022:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096940.542808:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096940.988946:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096941.571042:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096942.017709:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096942.592256:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096943.036622:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096943.616390:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096944.060509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766096944.444981:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417124864:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096944.446333:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417124928:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766096944.447837:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096944.447845:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096944.447849:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096944.447857:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1448 00000100:00080000:3.0:1766096944.447860:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096944.447865:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096944.447875:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096944.447897:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417124992/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766096944.447976:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124992:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766096944.448763:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096944.448785:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096944.455200:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417124928:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766096944.455920:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417124864:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:1.0:1766096944.457939:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:1.0:1766096944.467423:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766096944.467433:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766096944.467444:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417124992:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766096944.636628:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096945.084560:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096945.660547:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096946.108651:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096946.687226:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096947.132921:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096947.713770:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096948.157734:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096948.732868:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096949.180904:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766096949.566060:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096949.566066:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766096949.566069:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766096949.566079:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1453 00000100:00080000:1.0:1766096949.566082:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766096949.566101:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766096949.566112:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766096949.566129:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417125184/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766096949.566214:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125184:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096949.566309:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417125056:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096949.566347:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766096949.566390:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417125120:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096949.567241:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096949.567247:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096949.575547:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417125056:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096949.575640:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766096949.575645:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766096949.575649:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125184:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096949.576447:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417125120:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766096949.757840:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096950.204473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096950.780592:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096951.228818:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096951.807750:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096952.252497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096952.829227:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096953.276472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096953.854096:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096954.300511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766096954.687137:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096954.687144:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766096954.687147:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766096954.687157:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1458 00000100:00080000:1.0:1766096954.687161:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766096954.687166:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766096954.687176:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766096954.687192:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417125376/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766096954.687230:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096954.687233:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766096954.687254:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417125312:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096954.687341:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766096954.687347:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417125248:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096954.687409:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125376:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766096954.692687:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096954.692696:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096954.692705:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125376:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766096954.693332:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417125312:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096954.695567:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417125248:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766096954.879050:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096955.329274:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096955.900692:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766096956.351246:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096956.925657:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096957.372552:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096957.948808:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096958.398098:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096958.972654:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096959.420849:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766096959.804691:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096959.804700:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766096959.804705:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766096959.804718:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1463 00000100:00080000:2.0:1766096959.804723:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766096959.804730:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766096959.804744:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766096959.804763:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417125568/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766096959.804792:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096959.804795:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096959.805001:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417125440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766096959.805131:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766096959.805139:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417125504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096959.805255:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125568:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766096959.808176:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417125504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096959.811507:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766096959.811522:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766096959.811535:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125568:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766096959.816543:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417125440:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766096959.996665:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096960.445246:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096961.020595:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096961.468619:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096962.045807:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096962.492637:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096963.070604:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096963.516612:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096964.092654:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096964.540470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096964.924831:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096964.924840:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096964.924844:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096964.924858:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1468 00000100:00080000:0.0:1766096964.924863:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096964.924871:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096964.924884:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096964.924904:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52a4842 x1851885417125760/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766096964.924951:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417125632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766096964.924954:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096964.924961:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096964.925011:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417125696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766096964.925065:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125760:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096964.931991:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417125696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766096964.933229:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417125632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096964.933395:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096964.933400:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096964.933404:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125760:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766096965.118621:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096965.564692:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096966.140815:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096966.588860:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096967.165606:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096967.618841:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096968.188736:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096968.636609:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096969.212546:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096969.661478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766096970.050498:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096970.050505:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766096970.050508:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766096970.050518:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1474 00000100:00080000:2.0:1766096970.050521:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766096970.050526:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766096970.050536:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096970.050570:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417125888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096970.050599:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417125952/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766096970.050623:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096970.050626:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096970.050651:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766096970.050656:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417125824:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766096970.050737:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125952:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766096970.055545:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096970.055555:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096970.055564:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417125952:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766096970.056056:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417125824:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766096970.058511:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417125888:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766096970.237532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096970.685187:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096971.260532:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096971.708490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096972.284568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096972.732744:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096973.310697:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096973.756460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096974.332772:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096974.782456:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766096975.167414:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096975.167434:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096975.167437:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096975.167447:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1479 00000100:00080000:3.0:1766096975.167450:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096975.167456:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096975.167465:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096975.167479:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417126144/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766096975.167502:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096975.167505:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766096975.167528:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417126080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766096975.167608:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766096975.167613:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417126016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766096975.167695:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126144:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766096975.173313:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417126080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096975.179548:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096975.179558:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096975.179566:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126144:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766096975.179932:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417126016:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766096975.356583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096975.804479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096976.380559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096976.828457:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096977.404483:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766096977.852472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096978.433196:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096978.878634:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096979.452709:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096979.900546:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766096980.286394:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417126208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096980.288595:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766096980.288604:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766096980.288609:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766096980.288623:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1484 00000100:00080000:1.0:1766096980.288627:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766096980.288633:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766096980.288648:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766096980.288667:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417126336/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766096980.288683:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1766096980.289177:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126336:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766096980.290349:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766096980.290381:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417126272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766096980.291311:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766096980.300741:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417126208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766096980.303889:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417126272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766096980.303997:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766096980.304002:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766096980.304025:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126336:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766096980.497481:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096980.925109:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096981.501433:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096981.948542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096982.524578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096982.974472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096983.549710:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096983.996475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096984.572539:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096985.020484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766096985.404828:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417126400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096985.405032:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766096985.405038:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417126464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766096985.405144:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096985.405150:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766096985.405162:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766096985.405185:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1489 00000100:00080000:3.0:1766096985.405188:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766096985.405193:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766096985.405204:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766096985.405220:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417126528/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766096985.405446:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126528:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766096985.405957:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766096985.405961:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766096985.414068:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417126464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096985.414092:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417126400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096985.415410:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096985.415422:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096985.415430:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126528:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766096985.597820:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096986.044697:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096986.624634:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096987.070569:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096987.647248:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096988.092757:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096988.668670:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096989.119865:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096989.692642:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096990.140498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766096990.525903:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096990.525912:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766096990.525916:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766096990.525930:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1494 00000100:00080000:0.0:1766096990.525934:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766096990.525941:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766096990.525955:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766096990.525971:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417126720/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766096990.525987:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766096990.525990:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766096990.526210:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126720:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766096990.527684:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417126656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766096990.527774:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766096990.527782:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417126592:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096990.535666:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096990.535679:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096990.535691:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126720:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766096990.543385:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417126656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766096990.544932:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417126592:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766096990.717578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096991.167707:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096991.740554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096992.188465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096992.766504:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096993.212459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096993.789899:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096994.238798:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096994.812809:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096995.260461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766096995.648261:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096995.648268:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766096995.648272:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766096995.648282:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1499 00000100:00080000:2.0:1766096995.648285:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766096995.648327:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766096995.648338:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766096995.648397:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417126912/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766096995.648427:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766096995.648430:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766096995.648456:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417126848:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766096995.648535:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766096995.648541:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417126784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766096995.649373:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126912:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766096995.651842:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417126784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766096995.651877:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417126848:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766096995.651896:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766096995.651906:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766096995.651916:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417126912:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766096995.836939:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096996.287671:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096996.860489:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096997.308495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096997.884544:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096998.332469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766096998.908554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766096999.357224:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766096999.935208:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097000.386219:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097000.764543:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097000.764549:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097000.764553:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097000.764563:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1504 00000100:00080000:2.0:1766097000.764567:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097000.764572:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:3.0:1766097000.764582:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417126976:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:2.0:1766097000.764583:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097000.764601:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009119b13e x1851885417127104/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097000.764626:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097000.764629:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097000.764666:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417127040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097000.764679:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127104:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097000.773191:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417126976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097000.781454:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417127040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097000.782008:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097000.782018:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097000.782027:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127104:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097000.956627:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097001.404494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097001.980668:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097002.428641:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097003.005703:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097003.452785:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097004.028647:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097004.476513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097005.053445:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097005.503617:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097005.885008:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097005.885030:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097005.885035:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097005.885046:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1509 00000100:00080000:0.0:1766097005.885049:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097005.885058:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097005.885068:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097005.885086:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417127296/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766097005.885094:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417127168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097005.885122:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097005.885127:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097005.885177:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417127232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097005.885333:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127296:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097005.890893:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417127168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097005.891208:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417127232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097005.891227:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097005.891235:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097005.891241:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127296:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766097006.077875:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097006.524559:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097007.100978:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097007.548505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097008.125982:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097008.573388:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097009.150531:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097009.598754:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097010.172647:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097010.621388:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097011.004673:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097011.004680:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097011.004685:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097011.004699:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1514 00000100:00080000:0.0:1766097011.004723:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097011.004730:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097011.004743:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097011.004765:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417127488/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097011.004788:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097011.004792:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097011.004817:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417127424:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097011.004950:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097011.004955:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417127360:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097011.007696:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127488:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097011.009031:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417127424:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097011.009660:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417127360:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097011.012536:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097011.012552:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097011.012564:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127488:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097011.197813:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097011.649019:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097012.221155:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097012.669385:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097013.246249:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097013.692702:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097014.268905:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097014.716472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097015.292616:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097015.740527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097016.124884:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097016.124892:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097016.124895:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097016.124906:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1520 00000100:00080000:0.0:1766097016.124911:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097016.124919:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766097016.124919:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417127552:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1766097016.124934:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097016.124981:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417127680/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097016.125003:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097016.125020:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097016.125037:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097016.125047:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417127616:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097016.125277:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127680:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097016.131974:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417127616:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097016.132006:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417127552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097016.132337:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097016.132342:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097016.132346:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127680:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097016.319210:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097016.764913:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097017.340627:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097017.788555:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097018.366572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097018.812734:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097019.388760:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097019.836463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097020.412833:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097020.860466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097021.247121:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097021.247128:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097021.247131:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097021.247142:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1525 00000100:00080000:0.0:1766097021.247145:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097021.247150:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097021.247159:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097021.247173:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52a4842 x1851885417127872/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097021.247195:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097021.247198:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097021.247216:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417127808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097021.247300:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097021.247307:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417127744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097021.247313:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127872:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766097021.253220:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097021.253230:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097021.253239:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417127872:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097021.253316:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417127744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097021.253330:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417127808:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766097021.436519:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097021.884588:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097022.462830:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097022.908627:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097023.485427:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097023.932626:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097024.508602:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097024.957694:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097025.532640:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097025.980464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097026.365328:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097026.365335:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097026.365339:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097026.365349:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1530 00000100:00080000:0.0:1766097026.365431:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097026.365439:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097026.365449:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097026.365465:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000024975221 x1851885417128064/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097026.365491:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097026.365495:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097026.365521:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097026.365615:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097026.365623:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417127936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097026.365740:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766097026.379929:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097026.379937:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097026.379945:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097026.380404:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417127936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097026.380419:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128000:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097026.557142:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097027.005043:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097027.580572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097028.028477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097028.605135:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097029.055023:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097029.628521:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097030.076548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097030.654884:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097031.100471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097031.485757:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097031.485765:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097031.485768:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097031.485779:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1535 00000100:00080000:2.0:1766097031.485781:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097031.485787:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097031.485795:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097031.485811:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009119b13e x1851885417128256/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097031.485840:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097031.485843:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097031.485867:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417128192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097031.485945:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097031.485951:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417128128:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097031.486622:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128256:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097031.488318:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417128128:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097031.490333:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417128192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097031.494651:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097031.494662:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097031.494669:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128256:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097031.678497:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097032.124595:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097032.700715:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097033.151939:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097033.724578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097034.179517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097034.748578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097035.197439:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097035.772570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097036.220485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097036.604832:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097036.604841:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097036.604844:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097036.604854:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1540 00000100:00080000:0.0:1766097036.604857:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097036.604862:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097036.604872:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097036.604890:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000042205631 x1851885417128448/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097036.604915:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097036.604931:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097036.604956:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097036.605048:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097036.605055:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417128320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097036.605153:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766097036.610231:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097036.610247:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097036.610256:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097036.610742:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417128320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097036.610758:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128384:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766097036.797056:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097037.245524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097037.820635:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097038.268555:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097038.844610:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097039.292469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097039.868575:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097040.316714:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097040.892853:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097041.340488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097041.725827:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1766097041.725842:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417128512:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097041.725848:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097041.725851:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097041.725856:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1545 00000100:00080000:1.0:1766097041.725859:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097041.725865:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097041.725875:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097041.725889:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417128640/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097041.725914:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097041.725918:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097041.725945:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097041.725959:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128640:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097041.734900:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097041.736652:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097041.736659:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097041.736666:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128640:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097041.736866:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417128512:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097041.920413:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097042.364671:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097042.940963:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097043.389859:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097043.964851:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097044.412465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097044.992933:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097045.437417:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097046.014782:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097046.461515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097046.844567:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097046.844584:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097046.844587:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097046.844596:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1550 00000100:00080000:2.0:1766097046.844600:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097046.844605:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097046.844614:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097046.844627:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007b4dad67 x1851885417128832/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097046.844651:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097046.844654:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097046.844747:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128832:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097046.845271:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417128704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097046.845313:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097046.845318:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417128768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097046.848759:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417128704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097046.848862:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097046.848871:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097046.848888:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417128832:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097046.850708:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417128768:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097047.038868:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097047.488164:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097048.060599:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097048.508516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097049.085640:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097049.532995:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097050.110392:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097050.556477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097051.134244:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097051.580471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097051.964608:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097051.964616:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097051.964620:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097051.964631:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1555 00000100:00080000:1.0:1766097051.964635:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097051.964641:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097051.964652:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097051.964677:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000094c69f2b x1851885417129024/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097051.964712:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097051.964716:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097051.964733:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417128896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097051.964743:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097051.964930:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129024:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097051.971541:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417128896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097051.971787:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097051.971798:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097051.971807:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129024:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097051.975544:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417128960:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097052.158139:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097052.604483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097053.180538:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097053.633568:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097054.204549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097054.652484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097055.228582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097055.676481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097056.255596:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097056.700925:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097057.086005:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097057.086013:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097057.086017:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097057.086027:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1561 00000100:00080000:2.0:1766097057.086030:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097057.086036:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097057.086047:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097057.086076:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417129216/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097057.086100:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097057.086103:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097057.086131:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417129152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097057.086211:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097057.086218:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417129088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097057.089717:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129216:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097057.093625:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417129088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097057.093663:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417129152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097057.093856:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097057.093863:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097057.093868:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129216:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097057.279077:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097057.724796:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097058.302763:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097058.749112:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097059.325456:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097059.774935:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097060.348910:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097060.797171:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097061.374223:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097061.820467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097062.206267:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097062.206274:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097062.206277:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097062.206286:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1566 00000100:00080000:0.0:1766097062.206289:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097062.206295:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097062.206303:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097062.206316:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417129408/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097062.206334:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097062.206352:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097062.206559:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417129280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097062.206659:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097062.206666:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417129344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097062.207764:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129408:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097062.210457:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417129280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097062.210529:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097062.210534:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097062.210540:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129408:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097062.210556:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417129344:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097062.398452:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097062.844506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097063.421209:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097063.868529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097064.446374:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097064.892483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097065.468936:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097065.916861:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097066.493186:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097066.940578:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097067.324555:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097067.324562:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097067.324565:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766097067.324578:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417129472:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097067.324603:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1571 00000100:00080000:1.0:1766097067.324605:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097067.324611:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097067.324622:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097067.324637:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417129600/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097067.324665:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097067.324669:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097067.324676:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097067.324685:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417129536:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097067.324751:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129600:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097067.331000:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1766097067.331004:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417129536:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097067.331009:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097067.331017:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129600:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097067.331024:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417129472:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097067.516824:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097067.964474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097068.542028:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097068.989585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097069.566938:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097070.013273:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097070.588650:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097071.038828:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097071.617575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097072.060790:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097072.444684:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097072.444691:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097072.444696:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097072.444709:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1576 00000100:00080000:0.0:1766097072.444713:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097072.444721:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097072.444735:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097072.444754:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2c84c91 x1851885417129792/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097072.444786:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097072.444790:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097072.444810:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417129728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097072.444849:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417129664:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097072.444956:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129792:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097072.447714:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417129728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097072.448068:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097072.448073:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097072.448079:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129792:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097072.456069:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417129664:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097072.638908:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097073.084536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097073.660603:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097074.108465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097074.685385:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097075.132471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097075.708554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097076.156474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097076.735740:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097077.181596:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097077.565864:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097077.565872:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097077.565876:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097077.565886:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1581 00000100:00080000:2.0:1766097077.565888:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097077.565894:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097077.565903:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097077.565918:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417129984/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097077.565943:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097077.565946:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097077.566030:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129984:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097077.566386:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417129856:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097077.566432:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097077.566438:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417129920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097077.569656:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097077.569669:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097077.569679:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417129984:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097077.582993:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417129920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097077.583014:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417129856:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097077.756541:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097078.204481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097078.780915:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097079.228460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097079.804611:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097080.252897:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097080.828690:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097081.276522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097081.852604:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097082.300455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097082.684694:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097082.684702:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097082.684705:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097082.684716:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1586 00000100:00080000:2.0:1766097082.684719:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097082.684738:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097082.684748:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097082.684764:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417130176/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097082.684803:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097082.684807:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097082.684834:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417130112:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097082.684848:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130176:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097082.684922:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097082.684929:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417130048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097082.690910:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097082.690921:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097082.690933:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130176:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097082.691025:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417130048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097082.691040:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417130112:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766097082.876911:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097083.324494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097083.901753:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097084.348565:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097084.924567:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097085.372505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097085.948560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097086.396522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097086.972773:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097087.420478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097087.804658:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097087.804680:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097087.804683:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097087.804693:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1591 00000100:00100000:1.0:1766097087.804693:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417130240:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097087.804696:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097087.804702:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097087.804712:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097087.804727:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417130368/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097087.804761:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097087.804764:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097087.804787:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097087.804793:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417130304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097087.804852:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130368:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097087.813960:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417130304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097087.819227:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097087.819238:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097087.819248:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130368:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097087.821282:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417130240:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097088.002783:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097088.444622:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097089.020605:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097089.468749:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097090.045514:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097090.492536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097091.068590:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097091.516467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097092.094040:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097092.540474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097092.924692:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097092.924699:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097092.924703:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097092.924714:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1596 00000100:00080000:0.0:1766097092.924717:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097092.924722:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097092.924732:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097092.924748:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417130560/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097092.924775:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097092.924778:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097092.924803:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417130496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097092.924930:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097092.924938:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417130432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097092.925172:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130560:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097092.930003:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417130432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097092.939279:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417130496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097092.939581:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097092.939586:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097092.939604:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130560:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097093.118712:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097093.565232:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097094.140902:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097094.588506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097095.164539:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097095.612482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097096.189819:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097096.636448:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097097.213020:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097097.660764:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097098.046556:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097098.046564:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097098.046568:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097098.046578:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1602 00000100:00080000:3.0:1766097098.046582:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097098.046588:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097098.046598:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097098.046613:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbfb1ac7 x1851885417130752/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097098.046627:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097098.046629:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097098.046716:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417130624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097098.046792:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097098.046799:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417130688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097098.046981:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130752:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097098.057060:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417130688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:3.0:1766097098.058111:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:2.0:1766097098.059904:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417130624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097098.073767:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097098.073782:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097098.073798:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130752:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097098.237394:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097098.684931:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097099.260572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097099.708526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097100.284602:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097100.732482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097101.308992:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097101.757239:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097102.332738:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097102.780843:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097103.164991:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097103.164999:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097103.165003:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097103.165016:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1607 00000100:00080000:1.0:1766097103.165020:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097103.165027:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097103.165040:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097103.165056:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9051765 x1851885417130944/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097103.165078:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097103.165081:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097103.165152:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130944:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097103.165501:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417130816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097103.165547:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097103.165555:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417130880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097103.170899:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097103.170922:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097103.170935:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417130944:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097103.172206:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417130816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097103.172253:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417130880:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097103.357063:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097103.804747:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097104.380919:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097104.828474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097105.406040:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097105.854198:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097106.428957:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097106.876483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097107.453397:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097107.901302:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097108.284520:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097108.284537:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097108.284541:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097108.284551:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1612 00000100:00080000:1.0:1766097108.284554:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097108.284559:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097108.284568:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097108.284581:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417131136/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097108.284604:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097108.284607:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097108.284678:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417131072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097108.284764:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097108.284769:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417131008:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097108.288026:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131136:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097108.288308:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417131008:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097108.288325:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417131072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097108.291470:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097108.291481:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097108.291490:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131136:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097108.476660:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097108.925252:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097109.502346:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097109.948682:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097110.524689:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097110.973614:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097111.552728:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097111.996564:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097112.575162:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097113.020496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097113.405469:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097113.405475:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097113.405479:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097113.405489:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1617 00000100:00080000:3.0:1766097113.405492:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097113.405497:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766097113.405507:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417131200:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:3.0:1766097113.405507:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097113.405538:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417131328/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097113.405553:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097113.405557:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097113.405591:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417131264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097113.405716:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131328:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097113.417816:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417131264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097113.417878:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417131200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097113.420969:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097113.420979:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097113.421005:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131328:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097113.596577:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097114.044472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097114.620608:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097115.068476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097115.647956:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097116.092570:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097116.670494:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097117.116512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097117.694250:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097118.140626:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097118.524617:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097118.524625:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097118.524628:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097118.524638:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1622 00000100:00080000:3.0:1766097118.524642:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097118.524647:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097118.524656:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097118.524672:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417131520/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097118.524696:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097118.524699:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097118.524726:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417131456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097118.524729:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417131392:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097118.524830:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131520:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097118.533937:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417131392:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097118.534063:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097118.534068:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097118.534073:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131520:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097118.534169:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417131456:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097118.716836:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097119.164482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097119.740697:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097120.188686:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097120.764532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097121.216166:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097121.790303:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097122.236839:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097122.813066:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097123.260457:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097123.644568:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417131584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097123.644784:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097123.644790:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1766097123.644791:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1766097123.644793:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766097123.644797:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417131648:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097123.644798:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1627 00000100:00080000:3.0:1766097123.644801:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097123.644806:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097123.644816:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097123.644831:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417131712/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097123.644896:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131712:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766097123.645142:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097123.645146:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097123.648043:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417131584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097123.649550:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417131648:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097123.650210:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097123.650217:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097123.650223:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131712:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097123.837696:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097124.284487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097124.860718:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097125.308574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097125.885179:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097126.332768:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097126.908538:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097127.356474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097127.934751:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097128.380795:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097128.764641:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097128.764650:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097128.764653:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097128.764678:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1632 00000100:00080000:1.0:1766097128.764681:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097128.764688:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097128.764698:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097128.764716:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417131904/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097128.764743:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097128.764747:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097128.764836:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417131776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097128.764929:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097128.764936:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417131840:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097128.768760:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131904:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097128.769615:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417131776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097128.774681:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417131840:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097128.792910:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097128.792922:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097128.792931:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417131904:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766097128.956706:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097129.404466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097129.980891:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097130.428467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097131.004575:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097131.452467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097132.029509:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097132.476462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097133.053176:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097133.500579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097133.884502:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097133.884509:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097133.884512:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097133.884522:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1637 00000100:00080000:1.0:1766097133.884525:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097133.884530:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097133.884539:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097133.884552:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417132096/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097133.884616:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132096:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097133.884853:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132032:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097133.884892:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097133.884897:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417131968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097133.884965:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097133.884969:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097133.886794:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417131968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097133.886816:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132032:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097133.886895:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097133.886900:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097133.886905:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132096:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097134.076663:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097134.524484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097135.100785:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097135.548604:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097136.127691:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097136.573237:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097137.150804:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097137.596579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097138.172543:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097138.620482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097139.006784:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097139.006791:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097139.006794:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097139.006802:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1642 00000100:00080000:1.0:1766097139.006805:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097139.006810:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097139.006820:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097139.006833:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417132288/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097139.006856:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097139.006859:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097139.006938:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097139.007035:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097139.007041:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417132160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097139.010117:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097139.016491:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097139.017649:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097139.017657:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097139.017663:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097139.017852:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417132160:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097139.196571:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097139.644521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097140.220724:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097140.668481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097141.244560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097141.692478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097142.269598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097142.716722:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097143.292560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097143.740479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097144.124602:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097144.124610:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097144.124614:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097144.124638:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1648 00000100:00080000:0.0:1766097144.124641:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097144.124647:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097144.124657:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097144.124674:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000968f8523 x1851885417132480/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097144.124699:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097144.124702:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097144.124729:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097144.124915:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417132352:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097144.124973:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132480:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097144.131812:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097144.131823:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097144.131833:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132480:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097144.131911:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097144.131925:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417132352:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097144.317284:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097144.764485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097145.340527:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097145.789561:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097146.364922:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097146.812460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097147.388581:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097147.836480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097148.412653:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097148.860879:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097149.244680:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097149.244688:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097149.244691:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097149.244702:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1653 00000100:00080000:1.0:1766097149.244717:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097149.244724:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097149.244734:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097149.244748:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000174f4196 x1851885417132672/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097149.244772:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097149.244775:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097149.244849:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417132544:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097149.244928:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097149.244934:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417132608:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097149.245621:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132672:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097149.249644:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417132544:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097149.250749:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417132608:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097149.250870:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097149.250875:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097149.250880:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132672:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097149.437284:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097149.884675:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097150.466078:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097150.909991:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097151.485275:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097151.932713:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097152.508611:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097152.956597:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097153.532517:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097153.981028:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097154.364665:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097154.364672:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097154.364676:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097154.364690:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1658 00000100:00080000:0.0:1766097154.364694:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097154.364700:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097154.364712:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097154.364730:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004229f3d5 x1851885417132864/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097154.364756:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097154.364760:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097154.364793:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097154.365000:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097154.365946:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097154.365970:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417132736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097154.371448:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097154.371463:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097154.371475:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417132864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097154.371695:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417132800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097154.373484:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417132736:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097154.556572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097155.004508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097155.581126:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097156.028479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097156.604958:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097157.052470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097157.628727:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097158.077602:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097158.653780:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097159.100549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097159.484561:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097159.484568:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097159.484572:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097159.484582:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1663 00000100:00080000:2.0:1766097159.484585:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097159.484591:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097159.484601:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097159.484617:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417133056/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097159.484648:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097159.484651:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097159.484677:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417132992:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097159.484698:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417132928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097159.484966:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133056:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766097159.486847:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097159.486858:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097159.486865:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133056:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097159.488003:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417132992:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097159.489534:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417132928:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097159.676935:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097160.125274:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097160.700529:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097161.148457:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097161.724537:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097162.172593:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097162.748629:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097163.196503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097163.775819:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097164.220504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097164.604718:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097164.604725:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097164.604728:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097164.604737:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1668 00000100:00080000:1.0:1766097164.604740:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097164.604745:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097164.604754:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097164.604766:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002600e542 x1851885417133248/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097164.604802:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097164.604805:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097164.604884:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417133120:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766097164.604964:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097164.604969:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417133184:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097164.605174:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133248:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766097164.610581:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417133120:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766097164.612710:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097164.612721:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097164.612729:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133248:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766097164.614534:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417133184:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766097164.796565:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097165.244475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097165.820587:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097166.268473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097166.845102:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097167.294954:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097167.869194:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097168.316601:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097168.892689:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097169.340479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097169.725970:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097169.725977:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097169.725994:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1766097169.725996:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133312:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766097169.726003:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1673 00000100:00080000:2.0:1766097169.726007:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097169.726012:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097169.726022:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097169.726038:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000204d6b1d x1851885417133440/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097169.726055:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097169.726058:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097169.726086:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097169.726094:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133376:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766097169.726129:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133440:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766097169.730864:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133312:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097169.731898:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133376:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766097169.732434:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097169.732441:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097169.732447:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133440:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766097169.917175:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097170.364520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097170.941061:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097171.388662:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097171.968265:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097172.412602:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097172.990533:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097173.436503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097174.012592:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097174.460524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097174.847568:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097174.848304:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097174.848313:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097174.848770:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097174.848776:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097174.848780:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097174.848795:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1678 00000100:00080000:3.0:1766097174.848798:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097174.848803:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097174.848813:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097174.848857:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417133632/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097174.848910:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097174.849533:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097174.849537:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097174.855147:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097174.856522:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097174.858206:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097174.858219:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097174.858232:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133632:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097175.038959:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097175.484519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097176.062119:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097176.508917:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097177.085586:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097177.532535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097178.116852:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097178.556529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097179.132717:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097179.580503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097179.966756:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133696:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097179.967672:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097179.967684:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133760:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766097179.968955:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097179.968964:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097179.968966:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097179.968974:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1683 00000100:00080000:2.0:1766097179.968977:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097179.968996:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097179.969005:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097179.969028:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000204d6b1d x1851885417133824/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097179.969091:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133824:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1766097179.971150:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097179.971160:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097179.974137:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133760:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766097179.974153:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133696:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766097179.976252:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097179.976261:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097179.976269:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417133824:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766097180.156786:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097180.605081:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097181.180853:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097181.629486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097182.204862:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097182.654389:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097183.228604:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097183.676528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097184.252591:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097184.700499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097185.084665:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097185.084675:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097185.084681:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097185.084698:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1689 00000100:00080000:3.0:1766097185.084703:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097185.084722:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766097185.084723:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133888:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:3.0:1766097185.084739:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097185.084788:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417134016/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097185.084824:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097185.084829:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097185.084839:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097185.084852:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133952:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097185.085142:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134016:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097185.098829:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417133888:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097185.102130:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097185.102146:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097185.102158:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134016:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097185.102242:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417133952:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097185.280022:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097185.724512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097186.304128:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097186.748523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097187.325117:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097187.772515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097188.350791:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097188.796531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097189.373288:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097189.820548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097190.204636:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097190.204645:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097190.204649:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097190.204663:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1694 00000100:00080000:1.0:1766097190.204668:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097190.204683:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097190.204699:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097190.204719:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417134208/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097190.204746:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097190.204767:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097190.204860:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417134080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097190.204888:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097190.205640:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097190.205652:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417134144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097190.213142:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417134080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097190.213188:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417134144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097190.213271:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097190.213275:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097190.213279:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134208:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097190.397094:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097190.844660:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097191.420624:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097191.869829:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097192.446328:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097192.892684:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097193.468653:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097193.916714:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097194.494755:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097194.940990:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097195.324695:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417134272:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097195.325615:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417134336:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766097195.326348:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097195.326372:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097195.326375:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097195.326393:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1699 00000100:00080000:2.0:1766097195.326396:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097195.326401:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097195.326410:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097195.326426:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417134400/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097195.326445:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097195.326448:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097195.326531:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134400:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766097195.332393:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417134272:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766097195.335131:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417134336:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766097195.342462:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097195.342475:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097195.342506:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134400:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766097195.517471:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097195.964466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097196.540858:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097196.988508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097197.564937:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097198.012601:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097198.588693:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097199.036521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097199.612640:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097200.061271:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097200.444528:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097200.444537:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097200.444541:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097200.444554:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1704 00000100:00080000:0.0:1766097200.444559:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097200.444566:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097200.444579:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097200.444609:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023592e07 x1851885417134592/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097200.444634:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097200.444638:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097200.444675:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417134464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097200.444676:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134592:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097200.444801:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097200.444809:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417134528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097200.457249:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097200.457262:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097200.457274:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134592:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097200.458688:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417134528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097200.458708:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417134464:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097200.638351:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097201.084664:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097201.661642:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097202.108639:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097202.685024:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097203.132532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097203.710655:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097204.156471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097204.735259:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097205.180542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097205.564644:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097205.564653:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097205.564658:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097205.564672:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1709 00000100:00080000:1.0:1766097205.564676:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097205.564686:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766097205.564689:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417134720:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:1.0:1766097205.564701:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097205.564725:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007d99d644 x1851885417134784/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097205.564747:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097205.564750:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097205.564799:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097205.564813:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417134656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097205.567005:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417134656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097205.569128:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097205.576431:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417134720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097205.576570:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097205.576578:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097205.576583:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134784:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097205.756655:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097206.204510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097206.781935:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097207.228517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097207.804697:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097208.252496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097208.828962:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097209.276524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097209.855303:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097210.300483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097210.684694:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097210.684702:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097210.684707:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097210.684721:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1714 00000100:00080000:1.0:1766097210.684726:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097210.684733:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097210.684764:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097210.684783:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002600e542 x1851885417134976/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097210.684804:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097210.684807:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097210.684960:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417134848:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766097210.685070:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097210.685078:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417134912:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097210.687462:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134976:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766097210.688386:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417134848:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766097210.689124:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417134912:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766097210.690659:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097210.690668:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097210.690675:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417134976:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766097210.877390:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097211.324480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097211.900882:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097212.348606:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097212.925763:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097213.372478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097213.949942:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097214.396889:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097214.972562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097215.420523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097215.804628:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097215.804655:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097215.804658:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097215.804668:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1719 00000100:00080000:0.0:1766097215.804671:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097215.804676:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097215.804686:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097215.804703:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000292c14cb x1851885417135168/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097215.804726:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097215.804729:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097215.804757:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097215.804760:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135168:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097215.804869:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417135104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097215.808231:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097215.808240:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097215.808248:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135168:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097215.811029:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417135104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097215.811827:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135040:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097215.997471:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097216.444700:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097217.020811:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097217.469132:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097218.045289:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097218.492480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097219.071209:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097219.516477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097220.095537:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097220.544506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097220.925270:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135232:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097220.930547:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097220.930553:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097220.930557:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097220.930568:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1724 00000100:00080000:1.0:1766097220.930571:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097220.930577:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097220.930587:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097220.930604:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000baf3fdb7 x1851885417135360/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097220.930672:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135360:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097220.930712:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097220.930731:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417135296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097220.930857:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097220.930861:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097220.931041:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135232:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097220.938811:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417135296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097220.939140:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097220.939145:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097220.939162:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135360:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097221.119052:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097221.564679:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097222.140749:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097222.588565:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097223.164712:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097223.615959:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097224.192342:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097224.636474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097225.213474:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097225.660713:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097226.044551:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097226.045894:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417135488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097226.048871:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097226.048882:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097226.048885:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097226.048894:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1730 00000100:00080000:1.0:1766097226.048897:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097226.048904:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097226.048915:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097226.048942:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417135552/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097226.048967:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097226.048971:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097226.049064:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135552:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097226.055866:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097226.066581:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097226.066593:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097226.066602:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135552:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097226.067908:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417135488:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097226.236530:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097226.684470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097227.260577:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097227.708589:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097228.285147:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097228.732508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097229.308609:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766097229.757201:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097230.333736:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097230.780510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097231.164862:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1766097231.164869:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135616:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766097231.164870:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097231.164873:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097231.164878:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1735 00000100:00080000:0.0:1766097231.164880:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097231.164886:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097231.164895:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097231.164907:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417135744/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097231.164929:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097231.164932:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097231.164947:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417135680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766097231.164975:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135744:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766097231.169013:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417135680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766097231.169101:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097231.169106:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097231.169123:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135744:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766097231.171088:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417135616:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766097231.360216:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097231.804521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097232.383992:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097232.828802:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097233.405429:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097233.852690:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097234.430664:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097234.876481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097235.452554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097235.900470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097236.284607:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097236.284613:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097236.284617:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097236.284627:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1740 00000100:00080000:2.0:1766097236.284630:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097236.284636:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097236.284645:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097236.284659:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417135936/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097236.284684:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097236.284687:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097236.284714:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417135872:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097236.284795:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097236.284801:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417135808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097236.285148:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135936:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097236.297287:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417135808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097236.297310:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417135872:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097236.298666:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097236.298676:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097236.298683:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417135936:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097236.476545:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097236.925382:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097237.500527:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097237.948471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097238.524585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097238.972599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097239.548518:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097239.996494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097240.572529:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097241.022373:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097241.404640:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097241.404648:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097241.404651:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097241.404662:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1745 00000100:00080000:3.0:1766097241.404666:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097241.404671:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097241.404682:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097241.404696:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ebab33a4 x1851885417136128/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097241.404725:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097241.404728:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097241.405010:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136128:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097241.405117:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097241.405146:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097241.405151:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417136064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097241.421839:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417136064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097241.423011:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097241.423089:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097241.423095:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097241.423099:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136128:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097241.596582:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097242.044429:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097242.620622:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097243.068928:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097243.644607:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097244.095323:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097244.669710:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097245.118300:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097245.694019:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097246.140467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097246.525237:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097246.525244:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097246.525247:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097246.525296:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1750 00000100:00080000:3.0:1766097246.525299:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097246.525303:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097246.525313:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097246.525328:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ebab33a4 x1851885417136320/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097246.525348:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097246.525351:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097246.525454:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097246.525525:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097246.525529:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417136256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097246.526103:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136320:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766097246.534417:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097246.534428:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097246.534438:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136320:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097246.535976:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417136256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097246.535992:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136192:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097246.720778:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097247.164511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097247.741386:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097248.189126:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097248.764922:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097249.212480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097249.788590:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097250.237122:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097250.812573:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097251.260727:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097251.644588:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097251.644596:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097251.644600:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097251.644609:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1755 00000100:00080000:2.0:1766097251.644613:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097251.644619:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097251.644628:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097251.644643:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417136512/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097251.644667:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097251.644670:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097251.644694:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417136448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766097251.644777:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097251.644784:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417136384:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097251.645299:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136512:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:02020000:1.0:1766097251.650094:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:3.0:1766097251.652430:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417136384:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097251.652451:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417136448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766097251.664029:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097251.664041:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097251.664053:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136512:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766097251.837939:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097252.284472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097252.865075:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097253.308577:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097253.885657:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097254.332462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097254.908700:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097255.356494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097255.933138:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097256.380953:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097256.764768:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097256.764776:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097256.764798:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097256.764809:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1760 00000100:00080000:2.0:1766097256.764812:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097256.764818:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097256.764828:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097256.764844:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417136704/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097256.764871:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097256.764874:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097256.765251:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136576:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097256.765339:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097256.765345:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417136640:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097256.766374:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136704:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766097256.771307:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417136640:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097256.771327:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136576:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766097256.771727:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097256.771733:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097256.771738:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136704:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766097256.957156:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097257.404638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097257.980929:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097258.429290:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097259.004661:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097259.452506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097260.030675:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097260.500807:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097261.053266:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097261.564478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097261.884804:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097261.884810:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097261.884814:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097261.884823:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1765 00000100:00080000:3.0:1766097261.884826:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097261.884832:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097261.884842:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097261.884856:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417136896/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097261.884885:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097261.884888:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097261.884914:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417136832:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097261.884995:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097261.885000:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417136768:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097261.885232:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136896:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766097261.887284:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417136768:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766097261.887306:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417136832:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766097261.887488:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097261.887493:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097261.887497:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417136896:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766097262.077382:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097262.588471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097263.100663:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097263.612500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097264.124641:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097264.637784:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097265.151751:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097265.661127:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097266.175395:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097266.684911:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097267.004754:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097267.004762:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097267.004765:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097267.004775:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1770 00000100:00080000:3.0:1766097267.004778:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097267.004784:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097267.004808:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097267.004824:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417137088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097267.004855:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097267.004859:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097267.005169:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137088:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097267.006201:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097267.006251:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097267.006263:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417137024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097267.011286:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097267.011296:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097267.011306:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137088:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097267.015221:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417137024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097267.015238:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417136960:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097267.197660:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097267.708872:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097268.220823:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097268.732588:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097269.245452:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097269.756507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097270.269213:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097270.780499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097271.292644:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097271.804482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097272.124588:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097272.124595:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097272.124599:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097272.124608:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1776 00000100:00080000:2.0:1766097272.124611:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097272.124616:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097272.124640:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097272.124658:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417137280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097272.124681:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097272.124684:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097272.124709:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417137216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097272.124791:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097272.124797:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417137152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097272.125001:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137280:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097272.127815:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417137216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097272.132200:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417137152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097272.132464:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097272.132469:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097272.132486:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137280:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097272.317624:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097272.828500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097273.340586:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097273.852773:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097274.364543:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097274.878248:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097275.389123:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097275.901304:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097276.412553:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097276.924574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097277.244650:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097277.244656:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097277.244660:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097277.244670:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1781 00000100:00080000:0.0:1766097277.244673:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097277.244678:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097277.244688:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097277.244703:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417137472/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097277.244760:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137472:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766097277.244877:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097277.244881:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097277.244912:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417137344:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766097277.245680:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097277.245690:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417137408:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766097277.249937:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097277.249948:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097277.249958:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137472:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766097277.250964:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417137408:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766097277.250984:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417137344:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766097277.437804:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097277.948489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097278.460548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097278.972468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097279.484541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097279.996473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097280.509681:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097281.020480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097281.533064:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097282.045189:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097282.365183:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417137536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766097282.367158:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097282.367166:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097282.367170:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097282.367180:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1786 00000100:00080000:0.0:1766097282.367183:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097282.367188:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097282.367198:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097282.367215:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417137664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097282.367235:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097282.367239:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097282.367302:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417137600:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097282.367622:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137664:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1766097282.377428:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097282.377445:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097282.377458:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137664:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766097282.380556:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417137536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097282.383286:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417137600:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766097282.557674:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097283.069528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097283.580923:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097284.092510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097284.611988:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097285.116766:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097285.628609:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097286.141134:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097286.652530:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097287.164466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097287.487283:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097287.487290:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097287.487293:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097287.487303:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1791 00000100:00080000:2.0:1766097287.487306:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097287.487311:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097287.487329:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097287.487343:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417137856/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766097287.487424:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137856:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097287.487599:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417137728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097287.487632:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097287.487636:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417137792:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097287.487920:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097287.487924:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097287.495685:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417137792:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097287.495701:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417137728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097287.497520:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097287.497541:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097287.497549:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417137856:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097287.676736:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097288.188430:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097288.701493:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097289.212487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097289.726202:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097290.236487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097290.749924:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097291.260485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097291.772588:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097292.284486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097292.605132:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097292.605139:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097292.605142:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097292.605153:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1796 00000100:00080000:0.0:1766097292.605156:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097292.605161:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097292.605170:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097292.605185:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004229f3d5 x1851885417138048/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097292.605209:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097292.605212:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097292.605292:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417137920:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097292.605413:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097292.605419:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417137984:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097292.607156:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138048:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766097292.619877:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097292.619890:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097292.619901:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138048:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766097292.620084:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417137984:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097292.620096:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417137920:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766097292.796553:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097293.308585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097293.821082:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097294.335805:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097294.845738:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097295.363020:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097295.868743:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097296.380492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097296.893170:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097297.404488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097297.724594:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097297.724602:0:37:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097297.724605:0:37:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097297.724615:0:37:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1801 00000100:00080000:1.0:1766097297.724618:0:37:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097297.724624:0:37:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097297.724633:0:37:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097297.724646:0:37:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417138240/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097297.724668:0:37:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097297.724684:0:37:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097297.724709:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417138176:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097297.724799:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097297.724805:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417138112:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766097297.724930:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138240:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766097297.727539:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097297.727549:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097297.727559:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138240:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766097297.728172:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417138112:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766097297.729064:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417138176:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766097297.917304:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097298.428536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097298.942096:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097299.452476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097299.964544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097300.476468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097300.989026:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097301.500492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097302.012620:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097302.524523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097302.844669:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097302.844698:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417138304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097302.844829:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097302.844837:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1766097302.844844:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417138368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097302.844865:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097302.844876:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1806 00000100:00080000:2.0:1766097302.844880:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097302.844888:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097302.845077:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097302.845099:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417138432/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097302.845128:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097302.845131:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097302.845197:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138432:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097302.851058:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417138368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097302.851074:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417138304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097302.851168:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097302.851173:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097302.851178:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138432:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097303.037125:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097303.548474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097304.060735:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097304.572432:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097305.084942:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097305.597681:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097306.109129:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097306.620474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097307.133216:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097307.644484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097307.965947:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097307.965956:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097307.965959:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097307.965969:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1811 00000100:00080000:0.0:1766097307.965972:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097307.965977:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097307.965986:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097307.965999:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417138624/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097307.966024:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097307.966027:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097307.966113:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138624:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097307.966262:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417138496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097307.966292:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097307.966297:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417138560:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097307.973239:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097307.973250:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097307.973259:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138624:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097307.973388:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417138560:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097307.973403:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417138496:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097308.156550:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097308.668845:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097309.182982:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097309.692449:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097310.205453:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097310.716472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097311.232102:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097311.740525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097312.254035:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097312.764539:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097313.084533:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097313.084552:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097313.084556:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097313.084565:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1817 00000100:00080000:0.0:1766097313.084568:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097313.084574:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097313.084584:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097313.084599:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417138816/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097313.084631:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097313.084635:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097313.084661:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417138688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097313.084749:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097313.084757:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417138752:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097313.084873:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138816:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097313.089206:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417138752:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097313.089225:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417138688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097313.098054:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097313.098065:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097313.098075:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417138816:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097313.277170:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097313.788476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097314.300567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097314.812511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097315.324591:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097315.836501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097316.348546:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097316.860484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097317.372669:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097317.884467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097318.205671:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097318.205679:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097318.205683:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1766097318.205694:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417138880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097318.205700:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1822 00000100:00080000:0.0:1766097318.205702:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097318.205707:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097318.205716:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097318.205730:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417139008/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097318.205765:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097318.205769:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097318.205788:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417138944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097318.205815:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139008:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097318.213881:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417138880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097318.213946:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417138944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097318.215018:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097318.215024:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097318.215029:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139008:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097318.396623:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097318.908673:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097319.420626:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097319.932482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097320.444523:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097320.959395:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097321.470155:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097321.980509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097322.492536:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097323.004480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097323.324559:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097323.324568:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097323.324571:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097323.324581:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1827 00000100:00080000:1.0:1766097323.324584:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097323.324590:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097323.324600:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097323.324615:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417139200/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097323.324639:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097323.324643:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097323.324747:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417139072:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097323.324845:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097323.324851:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417139136:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097323.324957:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139200:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097323.335349:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417139072:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097323.336861:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417139136:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097323.342508:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097323.342517:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097323.342540:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139200:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097323.519905:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097324.028480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097324.540584:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097325.054289:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097325.564592:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097326.076485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097326.588570:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097327.101528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097327.614799:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097328.124427:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097328.444773:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417139264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097328.445204:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097328.445213:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417139328:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097328.445454:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097328.445459:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097328.445462:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097328.445467:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1832 00000100:00080000:2.0:1766097328.445469:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097328.445474:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097328.445484:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097328.445499:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417139392/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097328.445522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097328.445526:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097328.445560:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139392:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097328.450563:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417139264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097328.451630:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097328.452067:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097328.452077:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139392:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097328.452189:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417139328:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097328.638910:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097329.148476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097329.661072:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097330.174406:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097330.685074:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097331.197784:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097331.708554:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097332.221674:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097332.733391:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097333.244475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097333.564600:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097333.564607:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097333.564611:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097333.564655:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1837 00000100:00080000:0.0:1766097333.564659:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097333.564665:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097333.564675:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097333.564690:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d602f5b x1851885417139584/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097333.564713:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097333.564716:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097333.564742:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417139520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097333.564850:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097333.564857:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417139456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097333.565743:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097333.575193:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417139520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097333.575739:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097333.575749:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097333.575759:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097333.576026:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417139456:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097333.758944:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097334.268780:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097334.782386:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097335.292542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097335.805698:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097336.316476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097336.828840:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097337.340475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097337.854048:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097338.364479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097338.684671:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097338.684680:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097338.684685:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097338.684698:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1842 00000100:00080000:3.0:1766097338.684702:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097338.684710:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097338.684723:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097338.684740:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417139776/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097338.684770:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097338.684774:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097338.684823:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417139712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097338.684948:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097338.684954:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417139648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097338.685287:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139776:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097338.701388:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417139712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097338.701430:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417139648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097338.701537:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097338.701543:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097338.701547:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139776:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097338.876864:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097339.388464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097339.901052:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097340.412473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097340.925266:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097341.436505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097341.948712:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097342.460531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097342.972622:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097343.486251:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097343.804681:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417139840:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097343.805533:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097343.805544:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417139904:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097343.806149:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097343.806156:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097343.806159:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097343.806165:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1847 00000100:00080000:1.0:1766097343.806168:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097343.806174:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097343.806185:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097343.806205:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417139968/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097343.806257:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139968:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1766097343.807058:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097343.807064:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097343.808224:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417139840:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097343.812793:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417139904:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097343.815449:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097343.815461:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097343.815471:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417139968:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097343.997450:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097344.508479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097345.021888:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097345.532509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097346.046459:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097346.556485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097347.069386:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097347.580484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097348.092592:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097348.604480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097348.924722:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1766097348.924740:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417140096:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097348.924745:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097348.924750:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097348.924756:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1852 00000100:00080000:2.0:1766097348.924759:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097348.924765:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097348.924776:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097348.924793:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ba671fa0 x1851885417140160/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097348.924815:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097348.924819:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097348.924835:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417140032:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097348.924855:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140160:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1766097348.929740:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097348.929755:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097348.929769:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140160:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097348.929873:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417140096:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097348.932652:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417140032:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1766097349.116665:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097349.628553:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097350.140704:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097350.652606:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097351.165555:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097351.676600:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097352.189258:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097352.700624:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097353.216315:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097353.724469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097354.044670:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417140224:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097354.044825:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097354.044832:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097354.044836:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097354.044854:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1858 00000100:00080000:2.0:1766097354.044857:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097354.044864:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097354.044877:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097354.044901:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1d41706 x1851885417140352/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097354.044924:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097354.044946:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097354.044965:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417140288:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097354.045461:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140352:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1766097354.048983:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417140224:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097354.049446:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417140288:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097354.050306:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097354.050314:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097354.050321:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140352:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097354.236667:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097354.749637:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097355.261217:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097355.772519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097356.284805:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097356.796473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097357.308593:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097357.820481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097358.332614:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097358.844507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097359.164693:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417140416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097359.165895:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097359.165914:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097359.165918:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097359.165925:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1863 00000100:00080000:0.0:1766097359.165928:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097359.165934:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097359.165943:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097359.165959:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000030e2f50e x1851885417140544/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097359.166030:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140544:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097359.169736:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417140416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097359.169762:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097359.169772:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417140480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097359.170340:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097359.170346:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097359.170352:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140544:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766097359.170463:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097359.170468:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097359.176967:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417140480:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097359.356572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097359.869073:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097360.380932:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097360.892481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097361.405630:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097361.917099:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097362.428552:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097362.940488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097363.453097:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097363.964651:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097364.286015:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097364.286022:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097364.286025:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097364.286035:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1868 00000100:00080000:0.0:1766097364.286038:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097364.286043:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097364.286053:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097364.286067:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000042205631 x1851885417140736/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097364.286090:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097364.286093:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097364.286181:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417140608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097364.286274:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097364.286280:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417140672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097364.286444:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097364.291458:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417140608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097364.297425:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097364.297435:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097364.297445:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097364.298043:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417140672:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097364.476981:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097364.988473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097365.501061:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097366.012503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097366.524613:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097367.036511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097367.549425:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097368.063470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097368.572625:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097369.087227:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097369.404583:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417140800:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097369.406139:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097369.406146:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097369.406163:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097369.406170:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1873 00000100:00080000:0.0:1766097369.406173:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097369.406178:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097369.406189:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097369.406206:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417140928/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097369.406230:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097369.406234:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097369.406263:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140928:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097369.407067:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097369.407079:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417140864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097369.408965:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097369.408980:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097369.408989:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417140928:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097369.409067:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417140864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097369.409078:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417140800:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097369.597110:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097370.108476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097370.620634:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097371.132562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097371.644834:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097372.156784:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097372.669619:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097373.180469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097373.692632:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097374.205495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097374.526693:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097374.526700:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097374.526704:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097374.526715:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1878 00000100:00080000:2.0:1766097374.526718:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1766097374.526723:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417140992:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097374.526726:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097374.526739:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097374.526756:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007b4dad67 x1851885417141120/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097374.526777:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097374.526781:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097374.526803:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417141056:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097374.527179:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141120:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097374.530222:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417141056:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097374.532722:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097374.532733:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097374.532742:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141120:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097374.532861:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417140992:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097374.717086:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097375.229418:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097375.740648:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097376.252536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097376.767639:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097377.276521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097377.789204:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097378.300461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097378.812575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097379.324454:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097379.644580:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097379.644589:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097379.644593:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766097379.644600:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417141184:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097379.644601:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1883 00000100:00080000:1.0:1766097379.644605:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097379.644613:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097379.644623:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097379.644640:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417141312/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097379.644669:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097379.644673:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097379.644699:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417141248:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097379.644722:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141312:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1766097379.656489:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097379.656501:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097379.656511:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141312:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097379.656572:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417141184:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097379.657046:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417141248:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1766097379.837637:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097380.348476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097380.860552:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097381.372897:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097381.885218:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097382.397161:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097382.908579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097383.420466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097383.933192:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097384.444478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097384.765136:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097384.765144:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097384.765147:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097384.765158:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1888 00000100:00080000:2.0:1766097384.765161:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097384.765166:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097384.765177:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097384.765191:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417141504/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097384.765211:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097384.765228:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097384.765256:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417141440:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097384.765272:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141504:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097384.765332:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097384.765338:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417141376:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097384.767608:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097384.767618:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097384.767626:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141504:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097384.767715:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417141376:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097384.767728:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417141440:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097384.956619:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097385.468920:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097385.980570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097386.492487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097387.004825:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097387.516496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097388.028590:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097388.540473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097389.054376:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097389.564536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097389.886269:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097389.886277:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097389.886280:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097389.886289:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1893 00000100:00080000:3.0:1766097389.886292:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097389.886309:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097389.886319:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097389.886334:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417141696/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097389.886371:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097389.886374:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097389.886418:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141696:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097389.888461:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417141568:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097389.888834:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097389.888842:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417141632:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097389.898811:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097389.898823:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097389.898831:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141696:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097389.898917:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417141568:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097389.907759:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417141632:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097390.078946:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097390.588505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097391.101978:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097391.612459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097392.125419:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097392.636469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097393.148582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097393.660469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097394.173327:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097394.684508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097395.005342:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097395.005350:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097395.005385:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097395.005395:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1898 00000100:00080000:0.0:1766097395.005399:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097395.005405:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097395.005413:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097395.005443:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417141888/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097395.005457:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097395.005460:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097395.005559:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417141760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097395.005671:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097395.005679:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417141824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097395.005772:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141888:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097395.009611:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417141824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097395.009641:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417141760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097395.009726:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097395.009730:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097395.009734:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417141888:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097395.196560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097395.708479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097396.220799:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097396.732485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097397.244567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097397.756469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097398.271051:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097398.783932:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097399.292558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097399.804467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097400.125124:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097400.125132:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097400.125144:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097400.125155:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1904 00000100:00080000:1.0:1766097400.125157:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097400.125163:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097400.125173:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097400.125189:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417142080/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097400.125258:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142080:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097400.126426:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417141952:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097400.126741:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097400.126749:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417142016:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097400.128618:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097400.128626:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1766097400.129310:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097400.129326:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097400.129334:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142080:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097400.135087:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417142016:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097400.135107:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417141952:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097400.316574:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097400.828484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097401.342174:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097401.852738:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097402.364558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097402.876486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097403.388722:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097403.902396:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097404.412590:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097404.929317:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097405.244552:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097405.244560:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097405.244564:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097405.244575:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1909 00000100:00080000:1.0:1766097405.244578:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766097405.244586:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417142144:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097405.244586:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097405.244600:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097405.244627:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417142272/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097405.244654:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097405.244657:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097405.244697:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142272:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097405.244700:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417142208:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:02020000:2.0:1766097405.258534:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:0.0:1766097405.260539:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417142144:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097405.260606:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417142208:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097405.264722:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097405.264731:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097405.264741:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142272:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097405.436647:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097405.950195:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097406.463924:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097406.972494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097407.484940:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097407.996427:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097408.508584:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097409.020676:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097409.532625:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097410.044474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097410.364990:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417142336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097410.366127:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097410.366136:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097410.366140:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097410.366152:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1914 00000100:00080000:3.0:1766097410.366155:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097410.366162:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097410.366173:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097410.366192:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417142464/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097410.366218:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097410.366229:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417142400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097410.366259:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142464:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097410.366386:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097410.366390:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097410.371020:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417142336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097410.371652:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417142400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097410.372580:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097410.372588:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097410.372595:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142464:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097410.557514:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097411.068822:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097411.581968:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097412.093978:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097412.605604:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097413.116543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097413.630500:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097414.140839:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097414.655350:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097415.164474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097415.485251:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417142528:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097415.486270:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097415.486281:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417142592:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097415.487249:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097415.487256:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097415.487260:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097415.487265:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1919 00000100:00080000:1.0:1766097415.487268:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097415.487273:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097415.487282:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097415.487303:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417142656/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097415.487367:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142656:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1766097415.487926:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097415.487930:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097415.491491:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417142592:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097415.491507:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417142528:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1766097415.504530:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097415.504542:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097415.504553:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142656:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097415.689418:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097416.188467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097416.700548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097417.212451:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097417.724913:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097418.236474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097418.748582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097419.260473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097419.773680:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097420.284490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097420.604943:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097420.604952:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097420.604956:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1766097420.604958:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417142720:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097420.604966:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1924 00000100:00080000:2.0:1766097420.604969:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097420.604976:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:3.0:1766097420.605062:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097420.605069:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417142784:192.168.203.152@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1766097420.605131:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097420.605150:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417142848/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097420.605176:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097420.605180:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097420.605419:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142848:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097420.615784:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417142784:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097420.615909:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417142720:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097420.616331:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097420.616350:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097420.616377:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417142848:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097420.798123:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097421.308481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097421.821031:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097422.333374:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097422.845025:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097423.356606:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097423.869613:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097424.381031:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097424.892561:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097425.406402:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097425.725649:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097425.725657:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097425.725660:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097425.725671:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1929 00000100:00080000:2.0:1766097425.725674:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097425.725680:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097425.725690:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097425.725693:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417142976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097425.725706:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417143040/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097425.725730:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097425.725733:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097425.725766:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417142912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097425.726632:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143040:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097425.731762:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417142976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097425.736142:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417142912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097425.741088:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097425.741098:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097425.741107:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143040:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097425.921345:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097426.429089:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097426.940688:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097427.452481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097427.964506:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097428.476476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097428.988549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097429.500483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097430.015074:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097430.524470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097430.844583:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143104:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097430.846906:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097430.846934:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097430.846938:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097430.846949:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1934 00000100:00080000:2.0:1766097430.846953:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097430.846959:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:3.0:1766097430.846963:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00100000:2.0:1766097430.846971:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097430.846977:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143168:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097430.846991:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417143232/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097430.847019:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097430.847023:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097430.847073:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143232:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097430.854257:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143168:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097430.854275:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143104:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097430.854350:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097430.854390:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097430.854394:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143232:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097431.037378:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097431.548543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097432.061773:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097432.572703:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097433.085864:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097433.596471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097434.108553:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097434.621945:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097435.132540:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097435.644447:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097435.968452:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097435.968474:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097435.968477:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097435.968487:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1939 00000100:00080000:3.0:1766097435.968490:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097435.968495:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097435.968504:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097435.968519:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417143424/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097435.968546:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097435.968550:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097435.968619:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097435.968704:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097435.968710:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097435.968972:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143424:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097435.973150:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097435.973296:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097435.973300:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097435.973304:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143424:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097435.975766:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143360:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097436.156898:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097436.668485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097437.181475:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097437.692477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097438.205260:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097438.716507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097439.228577:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097439.743235:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097440.257896:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097440.764525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097441.084730:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417143488:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097441.085467:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097441.085473:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097441.085491:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097441.085500:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1945 00000100:00080000:1.0:1766097441.085503:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097441.085509:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097441.085519:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097441.085537:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417143616/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097441.085563:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097441.085567:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097441.085601:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417143552:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097441.085632:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143616:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097441.093494:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417143488:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097441.093629:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417143552:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097441.094971:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097441.094981:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097441.094987:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143616:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097441.276554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097441.788470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097442.300582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097442.812459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097443.324767:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097443.838746:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097444.348875:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097444.860475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097445.372673:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097445.884480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097446.204648:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143680:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097446.204971:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097446.204978:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143744:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097446.207264:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097446.207276:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097446.207280:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097446.207292:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1950 00000100:00080000:2.0:1766097446.207296:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 6s 00000100:00080000:2.0:1766097446.207303:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097446.207315:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097446.207344:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417143808/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766097446.207443:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143808:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1766097446.208256:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097446.208263:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097446.212246:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143744:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097446.212264:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143680:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097446.213053:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097446.213061:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097446.213066:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417143808:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097446.396542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097446.908502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097447.420525:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097447.932446:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097448.444541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097448.956475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097449.468570:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097449.980482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097450.492847:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097451.006531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097451.326747:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097451.326755:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097451.326758:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097451.326776:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1955 00000100:00080000:2.0:1766097451.326780:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:2.0:1766097451.326786:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:3.0:1766097451.326790:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143872:192.168.203.152@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1766097451.326795:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097451.326810:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417144000/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097451.326888:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097451.326896:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144000:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1766097451.326908:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143936:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097451.327798:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097451.327805:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097451.334289:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417143872:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097451.336089:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097451.336099:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097451.336122:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144000:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1766097451.336404:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417143936:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097451.516981:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097452.033996:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097452.540564:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097453.052491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097453.569699:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097454.078540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097454.590677:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097455.100659:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097455.620634:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097456.124483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097456.447493:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097456.447500:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097456.447504:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097456.447515:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1960 00000100:00080000:1.0:1766097456.447518:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1766097456.447524:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097456.447533:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097456.447548:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417144192/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097456.447566:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097456.447570:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097456.447590:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417144064:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097456.447672:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097456.447678:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417144128:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097456.447957:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144192:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1766097456.452785:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097456.452795:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097456.452804:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144192:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097456.452917:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417144064:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097456.453990:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417144128:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1766097456.636581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097457.149749:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097457.661098:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097458.172506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097458.684544:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097459.196479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097459.708569:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097460.220509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097460.733012:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097461.244499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097461.564569:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097461.564577:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097461.564597:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097461.564606:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1965 00000100:00080000:1.0:1766097461.564611:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097461.564617:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097461.564627:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097461.564654:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417144384/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097461.564684:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097461.564694:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097461.564719:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417144320:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097461.564807:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097461.564813:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417144256:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097461.567777:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144384:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097461.569809:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417144320:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097461.573496:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417144256:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097461.573834:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097461.573839:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097461.573843:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144384:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097461.756582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097462.268453:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097462.780552:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097463.292947:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097463.804583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097464.316478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097464.828829:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097465.340466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097465.853597:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097466.366783:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097466.685481:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097466.685488:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097466.685491:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097466.685501:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1970 00000100:00080000:2.0:1766097466.685504:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766097466.685509:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097466.685518:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097466.685532:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1f2cb20 x1851885417144576/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097466.685572:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097466.685576:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097466.685660:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417144448:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097466.686062:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144576:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1766097466.686632:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097466.686639:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417144512:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097466.696669:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417144448:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097466.697657:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097466.697664:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097466.697669:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144576:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1766097466.698946:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417144512:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097466.876496:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097467.388475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097467.904870:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097468.414120:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097468.925338:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097469.439467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097469.950175:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097470.460479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097470.975679:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097471.484969:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097471.805128:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417144640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097471.805308:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097471.805315:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417144704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097471.805494:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097471.805499:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097471.805502:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097471.805507:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1975 00000100:00080000:0.0:1766097471.805510:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097471.805515:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097471.805525:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097471.805541:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005ed5ddd4 x1851885417144768/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097471.805564:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097471.805567:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097471.805597:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144768:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097471.813077:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417144704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097471.813096:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417144640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097471.813192:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097471.813196:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097471.813200:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144768:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097471.996570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097472.508478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097473.023694:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097473.534724:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097474.044704:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097474.556469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097475.069290:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097475.580472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097476.092772:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097476.604508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097476.924677:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097476.924685:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097476.924689:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097476.924699:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1980 00000100:00080000:1.0:1766097476.924714:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097476.924720:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097476.924731:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097476.924746:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f7eb885 x1851885417144960/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097476.924775:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097476.924778:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097476.924798:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417144896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097476.924875:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097476.924881:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417144832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097476.924995:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144960:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097476.936031:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417144896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097476.943234:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417144832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097476.943625:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097476.943631:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097476.943636:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417144960:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097477.117342:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097477.631255:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097478.141483:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097478.652514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097479.164595:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097479.676493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097480.189391:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097480.701195:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097481.218794:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097481.724468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097482.045597:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097482.045604:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097482.045608:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097482.045617:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1986 00000100:00080000:3.0:1766097482.045621:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766097482.045628:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097482.045637:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097482.045652:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417145152/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097482.045675:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097482.045679:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097482.045748:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145152:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097482.047583:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417145088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097482.047835:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097482.047843:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417145024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097482.055310:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097482.055321:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097482.055328:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145152:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097482.057926:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417145024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097482.057953:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417145088:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097482.236764:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097482.748800:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097483.261141:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097483.772478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097484.284577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097484.796537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097485.311909:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097485.821421:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097486.332601:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097486.848971:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097487.166135:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097487.166142:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097487.166158:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097487.166169:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1991 00000100:00080000:0.0:1766097487.166172:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097487.166178:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097487.166188:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097487.166203:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000024975221 x1851885417145344/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097487.166229:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097487.166233:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097487.166336:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145344:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097487.166701:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417145216:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097487.166749:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097487.166755:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417145280:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097487.171276:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417145216:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097487.175788:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417145280:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097487.179683:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097487.179694:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097487.179718:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145344:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097487.357623:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097487.868481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097488.382718:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097488.906532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097489.407404:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097489.916527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097490.428593:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097490.940513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097491.453931:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097491.964491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097492.286412:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097492.286418:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097492.286421:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097492.286430:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 1996 00000100:00080000:0.0:1766097492.286433:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097492.286439:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097492.286447:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097492.286459:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000512a78fc x1851885417145536/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097492.286505:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097492.286508:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097492.286526:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417145472:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097492.286594:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097492.286598:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417145408:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1766097492.287858:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145536:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097492.292981:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417145408:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097492.293006:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417145472:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097492.304844:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097492.304855:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097492.304864:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145536:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097492.476613:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097492.993613:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097493.501741:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097494.013439:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097494.525385:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097495.036677:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097495.548572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097496.060717:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097496.577951:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097497.084481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097497.404634:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097497.404642:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097497.404645:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097497.404655:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2001 00000100:00080000:1.0:1766097497.404659:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097497.404664:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766097497.404823:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417145600:192.168.203.152@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1766097497.406167:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097497.406197:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417145728/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097497.406255:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145728:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097497.406766:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097497.406774:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417145664:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097497.407131:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097497.407135:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097497.409313:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417145600:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097497.410849:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097497.410860:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097497.410867:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145728:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097497.411150:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417145664:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097497.597606:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097498.108500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097498.620642:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097499.132483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097499.644567:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097500.156528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097500.668567:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097501.180498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097501.693840:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097502.205131:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097502.525210:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097502.525217:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097502.525220:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097502.525237:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2006 00000100:00080000:3.0:1766097502.525240:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766097502.525247:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097502.525257:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097502.525279:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006796bf6d x1851885417145920/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097502.525309:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097502.525313:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097502.525347:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417145856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097502.525379:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145920:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097502.525521:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417145792:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097502.528952:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417145856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097502.530340:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097502.530348:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097502.530380:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417145920:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097502.530576:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417145792:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097502.716589:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097503.228512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097503.740967:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097504.252485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097504.765749:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097505.276495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097505.790402:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097506.300492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097506.812599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097507.325214:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097507.644681:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417145984:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097507.645687:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097507.645697:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417146048:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097507.646688:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097507.646711:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097507.646714:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097507.646721:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2011 00000100:00080000:0.0:1766097507.646725:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097507.646730:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097507.646742:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097507.647240:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52a4842 x1851885417146112/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097507.647269:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097507.647272:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097507.647316:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146112:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097507.647742:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417145984:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097507.652867:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097507.653395:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097507.653406:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146112:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097507.656123:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417146048:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097507.836766:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097508.348528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097508.860577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097509.373774:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097509.889708:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097510.396490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097510.909225:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097511.420538:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097511.933483:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097512.444485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097512.766593:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097512.766619:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417146176:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097512.766720:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097512.766724:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097512.766727:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097512.766732:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2016 00000100:00080000:1.0:1766097512.766735:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097512.766740:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766097512.766750:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417146240:192.168.203.152@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1766097512.766787:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097512.766803:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417146304/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097512.766829:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097512.766832:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097512.768612:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146304:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:3.0:1766097512.776483:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097512.776493:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097512.776513:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146304:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097512.776696:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417146240:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097512.776708:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417146176:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1766097512.956566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097513.468738:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097513.982890:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097514.492476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097515.005698:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097515.516460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097516.029283:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097516.541762:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097517.055984:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097517.564548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097517.884903:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097517.884911:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097517.884915:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097517.884926:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2021 00000100:00080000:1.0:1766097517.884929:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1766097517.884937:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417146368:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097517.884951:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097517.884962:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097517.884980:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002600e542 x1851885417146496/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097517.885006:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097517.885009:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097517.885031:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417146432:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097517.885937:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146496:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097517.887721:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417146368:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097517.892144:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417146432:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1766097517.892711:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097517.892718:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097517.892722:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146496:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097518.076665:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097518.588536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097519.105610:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097519.612477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097520.124544:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097520.636475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097521.148556:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097521.660548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097522.176328:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097522.684569:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097523.004609:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417146560:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097523.006967:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097523.006974:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097523.006978:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097523.006987:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2026 00000100:00080000:3.0:1766097523.006990:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766097523.006996:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097523.007007:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097523.007023:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417146688/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097523.007048:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097523.007052:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097523.007083:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097523.007110:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417146624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097523.007407:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146688:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766097523.016384:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097523.016395:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097523.016404:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146688:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097523.016522:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417146624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097523.016534:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417146560:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097523.196656:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097523.708487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097524.222921:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097524.733029:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097525.244775:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097525.756444:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097526.273542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097526.780511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097527.294276:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097527.804488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097528.124574:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097528.124581:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097528.124584:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097528.124593:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2032 00000100:00080000:3.0:1766097528.124596:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766097528.124602:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097528.124611:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097528.124615:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417146752:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097528.124625:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417146880/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097528.124646:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097528.124649:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097528.124667:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417146816:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097528.124769:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146880:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097528.130714:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417146816:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097528.130771:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417146752:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097528.131279:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097528.131285:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097528.131296:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417146880:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097528.316537:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097528.829308:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097529.341096:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097529.852495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097530.365141:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097530.876492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097531.389911:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097531.903262:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097532.413803:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097532.924482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097533.244570:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097533.244577:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097533.244580:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097533.244591:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2037 00000100:00080000:0.0:1766097533.244593:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097533.244599:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097533.244621:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097533.244638:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417147072/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097533.244660:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097533.244663:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097533.244734:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417146944:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097533.244820:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097533.244826:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417147008:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097533.244901:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147072:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097533.248293:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417147008:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097533.254955:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417146944:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1766097533.255251:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097533.255257:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097533.255261:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147072:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097533.440250:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097533.948471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097534.460524:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097534.972486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097535.484582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097535.999297:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097536.508890:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097537.021461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097537.533177:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097538.045767:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097538.364539:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097538.364546:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097538.364550:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097538.364559:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2042 00000100:00080000:0.0:1766097538.364562:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097538.364567:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766097538.364571:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417147200:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1766097538.364576:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097538.364590:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417147264/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097538.364613:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097538.364616:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097538.364660:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417147136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097538.365401:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147264:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097538.367905:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417147136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097538.368719:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417147200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097538.373525:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097538.373536:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097538.373544:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147264:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097538.558851:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097539.068689:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097539.580580:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097540.093066:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097540.605348:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097541.116788:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097541.630025:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097542.141336:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097542.653434:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097543.164479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097543.484650:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417147328:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097543.487012:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097543.487034:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417147392:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097543.490189:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417147328:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097543.490690:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417147392:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097543.494148:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097543.494158:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097543.494161:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097543.494171:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2047 00000100:00080000:1.0:1766097543.494174:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097543.494180:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097543.494190:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097543.494220:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000653e2b14 x1851885417147456/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097543.494244:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097543.494248:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097543.494279:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147456:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1766097543.503070:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097543.503081:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097543.503099:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147456:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097543.676562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097544.188552:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097544.700576:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097545.212463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097545.724580:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097546.236472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097546.750113:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097547.260691:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097547.772850:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097548.284509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097548.604583:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097548.604589:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097548.604592:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097548.604603:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2052 00000100:00080000:0.0:1766097548.604606:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766097548.604611:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097548.604620:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097548.604634:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417147648/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097548.604657:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097548.604660:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097548.604697:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417147584:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097548.604787:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097548.604794:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417147520:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097548.604969:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147648:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097548.612395:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417147584:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097548.612428:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417147520:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1766097548.617128:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097548.617139:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097548.617149:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147648:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097548.798973:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097549.308531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097549.820836:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097550.332480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097550.844737:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097551.356572:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097551.871102:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097552.380508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097552.894230:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097553.404687:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097553.724764:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097553.724846:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097553.724998:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766097553.725953:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417147712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097553.726066:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097553.726074:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417147776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097553.727663:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2057 00000100:00080000:3.0:1766097553.727669:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766097553.727676:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097553.727688:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097553.727717:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417147840/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097553.727788:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147840:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097553.728908:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097553.728917:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766097553.734145:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097553.734633:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097553.734768:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417147840:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097553.735495:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417147712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097553.735539:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417147776:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097553.916920:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097554.428621:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097554.941260:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097555.454021:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097555.964678:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097556.476471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097556.988535:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097557.500866:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097558.013858:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097558.524460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097558.844595:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097558.844602:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097558.844606:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097558.844617:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2062 00000100:00080000:1.0:1766097558.844620:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097558.844625:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097558.844635:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097558.844665:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417148032/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097558.844689:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097558.844692:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097558.844779:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417147904:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097558.844867:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097558.844872:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417147968:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097558.845380:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148032:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097558.853665:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417147904:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097558.857604:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417147968:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:02020000:3.0:1766097558.857711:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:3.0:1766097558.861709:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097558.861722:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097558.861752:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148032:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097559.036536:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097559.553349:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097560.062103:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097560.572468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097561.085325:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097561.596536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097562.108514:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097562.620481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097563.132567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097563.646859:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097563.964491:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417148096:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097563.964690:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097563.964695:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097563.964698:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097563.964703:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2067 00000100:00080000:1.0:1766097563.964706:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097563.964711:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097563.964720:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097563.964733:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007d99d644 x1851885417148224/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097563.964748:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097563.964751:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097563.964763:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148224:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1766097563.964845:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417148160:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097563.967914:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417148096:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097563.967961:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417148160:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097563.969128:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097563.969136:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097563.969155:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148224:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097564.156559:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097564.668518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097565.185387:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097565.693072:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097566.206132:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097566.716721:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097567.229082:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097567.740482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097568.252541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097568.765109:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097569.084563:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097569.084570:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097569.084573:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097569.084597:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2073 00000100:00080000:1.0:1766097569.084601:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097569.084606:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097569.084617:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097569.084634:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417148416/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097569.084663:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097569.084667:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097569.084698:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417148352:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097569.084788:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097569.084794:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417148288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097569.087918:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148416:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097569.092040:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097569.092051:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097569.092061:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148416:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097569.095044:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417148288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097569.095066:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417148352:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097569.276587:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097569.788533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097570.300565:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097570.812982:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097571.324566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097571.836479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097572.348606:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097572.862995:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097573.372568:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097573.885979:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097574.205728:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097574.205734:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097574.205737:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097574.205746:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2078 00000100:00080000:2.0:1766097574.205749:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766097574.205755:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097574.205762:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097574.205775:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417148608/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097574.205795:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417148480:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1766097574.205799:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097574.205803:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097574.205824:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417148544:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097574.205905:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148608:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1766097574.209901:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417148480:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097574.211070:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417148544:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1766097574.212117:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097574.212125:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097574.212131:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148608:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097574.397914:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097574.908508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097575.420561:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097575.934794:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097576.447086:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097576.956493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097577.468550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097577.980482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097578.492568:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097579.006841:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097579.324752:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097579.324761:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097579.324777:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097579.324787:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2083 00000100:00080000:3.0:1766097579.324790:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766097579.324796:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097579.324806:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097579.324852:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417148800/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097579.324874:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097579.324877:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097579.324884:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417148672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097579.324897:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417148736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097579.325213:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148800:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097579.333492:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417148672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097579.338882:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097579.338892:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097579.338901:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148800:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097579.342875:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417148736:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097579.522349:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097580.028478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097580.543316:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097581.052586:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097581.564536:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097582.076473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097582.588498:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097583.100484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097583.612808:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097584.125704:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097584.446582:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097584.446590:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097584.446594:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097584.446617:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2088 00000100:00080000:1.0:1766097584.446620:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097584.446626:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097584.446636:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097584.446652:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000174f4196 x1851885417148992/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097584.446920:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148992:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097584.447057:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417148864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097584.447094:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097584.447100:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417148928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097584.447182:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097584.447186:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097584.452528:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417148864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097584.453502:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097584.453510:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097584.453517:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417148992:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097584.453607:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417148928:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097584.637548:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097585.148468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097585.661175:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097586.172698:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097586.685067:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097587.196918:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097587.708716:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097588.220640:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097588.732584:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097589.244473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097589.564687:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149056:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097589.567923:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097589.567930:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097589.567934:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097589.567944:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2093 00000100:00080000:1.0:1766097589.567947:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766097589.567952:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097589.567962:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097589.567976:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417149184/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097589.567994:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097589.567997:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097589.568089:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417149120:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097589.568388:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149184:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097589.568544:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149056:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097589.578384:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417149120:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1766097589.581143:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097589.581157:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097589.581186:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149184:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1766097589.758860:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097590.268534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097590.781240:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097591.292519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097591.805013:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097592.316875:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097592.829224:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097593.341898:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097593.856279:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097594.364581:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097594.684599:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097594.684625:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097594.684629:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766097594.684644:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149248:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097594.684654:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2098 00000100:00080000:1.0:1766097594.684658:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097594.684666:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097594.684679:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097594.684701:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417149376/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097594.684739:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097594.684743:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097594.684770:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417149312:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097594.684891:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149376:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097594.693408:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149248:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1766097594.693416:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417149312:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1766097594.693535:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097594.693542:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097594.693547:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149376:192.168.203.152@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1766097594.877227:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097595.390784:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097595.900663:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097596.412779:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097596.925165:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097597.438648:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097597.948722:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097598.460508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097598.972588:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097599.484438:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097599.804552:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097599.804559:0:19925:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097599.804563:0:19925:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097599.804572:0:19925:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2103 00000100:00080000:1.0:1766097599.804575:0:19925:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766097599.804581:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149440:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1766097599.804581:0:19925:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097599.804589:0:19925:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097599.804608:0:19925:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417149568/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097599.804643:0:19925:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097599.804647:0:19925:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097599.804660:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097599.804666:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417149504:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1766097599.805808:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149568:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00080000:3.0:1766097599.814380:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097599.814391:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097599.814400:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149568:192.168.203.152@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1766097599.814912:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149440:192.168.203.152@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1766097599.814940:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417149504:192.168.203.152@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1766097599.996560:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097600.508474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097601.020729:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097601.534592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097602.044595:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097602.556476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097603.069767:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097603.580470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097604.096694:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097604.604513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097604.924610:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097604.926155:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097604.926162:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097604.926167:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097604.926175:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2108 00000100:00080000:0.0:1766097604.926179:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097604.926184:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097604.926193:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097604.926211:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417149760/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097604.926280:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149760:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097604.927475:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417149696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097604.927719:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097604.927724:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097604.932032:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417149696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097604.932222:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417149632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097604.932394:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097604.932398:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097604.932402:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149760:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097605.117284:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097605.628590:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097606.141975:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097606.657585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097607.166125:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097607.676495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097608.188513:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097608.700504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097609.212531:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097609.724515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097610.044770:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097610.044793:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097610.044796:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097610.044807:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2114 00000100:00080000:3.0:1766097610.044810:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097610.044816:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766097610.044826:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417149824:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:3.0:1766097610.044826:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097610.044846:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006796bf6d x1851885417149952/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097610.044874:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097610.044878:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097610.044902:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417149888:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097610.045182:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149952:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097610.052070:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417149824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097610.052777:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097610.052783:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097610.052789:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417149952:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097610.053348:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417149888:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097610.237220:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097610.748775:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097611.263925:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097611.772517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097612.285450:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097612.799201:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097613.311792:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097613.820546:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097614.332583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097614.844515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097615.165064:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097615.165072:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097615.165075:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097615.165084:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2119 00000100:00080000:0.0:1766097615.165088:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097615.165093:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097615.165102:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097615.165133:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417150144/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097615.165152:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097615.165155:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097615.165182:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417150080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097615.165274:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097615.165280:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417150016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097615.166284:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150144:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097615.171381:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417150080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097615.171418:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417150016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097615.173008:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097615.173016:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097615.173023:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150144:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097615.356673:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097615.869268:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097616.381080:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766097616.892590:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097617.404976:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097617.916478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097618.428620:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097618.940921:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097619.454424:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097619.968724:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097620.284647:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417150208:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097620.284810:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097620.284817:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417150272:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097620.285932:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097620.285942:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097620.285946:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097620.285954:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2124 00000100:00080000:2.0:1766097620.285958:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097620.285963:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097620.285974:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097620.285999:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417150336/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097620.286025:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097620.286029:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097620.286060:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150336:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097620.289333:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417150272:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097620.289350:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417150208:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097620.289650:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097620.289655:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097620.289660:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150336:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766097620.476579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097620.988452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097621.501554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097622.014753:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097622.525832:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097623.038109:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097623.551029:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097624.060473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097624.577725:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097625.084500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097625.404598:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097625.404605:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097625.404609:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097625.404619:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2129 00000100:00080000:0.0:1766097625.404621:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097625.404627:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097625.404651:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097625.404667:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000968f8523 x1851885417150528/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097625.404692:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097625.404696:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097625.404719:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417150464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097625.404725:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150528:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097625.404833:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417150400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097625.414987:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417150464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097625.415405:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097625.415410:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097625.415415:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150528:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097625.415525:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417150400:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097625.596540:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097626.108461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097626.620555:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097627.132482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097627.646725:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097628.158603:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097628.669720:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097629.180572:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097629.693729:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097630.205405:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097630.524879:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097630.524885:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097630.524889:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097630.524899:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2134 00000100:00080000:3.0:1766097630.524902:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097630.524907:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097630.524917:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097630.524930:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417150720/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097630.524953:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097630.524957:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097630.524971:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417150656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097630.524982:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150720:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097630.524991:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417150592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097630.532671:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417150656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097630.532720:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417150592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097630.533194:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097630.533200:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097630.533205:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150720:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097630.716708:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097631.228532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097631.740594:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097632.252487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097632.764594:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097633.276505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097633.789100:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097634.300516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097634.812603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097635.325336:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097635.644612:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097635.644619:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097635.644624:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097635.644635:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2139 00000100:00080000:0.0:1766097635.644639:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097635.644645:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766097635.644652:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417150784:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1766097635.644661:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097635.644681:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023592e07 x1851885417150912/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097635.644703:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097635.644707:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097635.644729:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150912:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097635.644755:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097635.644765:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417150848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097635.649652:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417150848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097635.649684:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417150784:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097635.649749:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097635.649753:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097635.649757:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417150912:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097635.836558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097636.349459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097636.863753:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097637.373324:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097637.884931:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097638.396483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097638.908568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097639.420504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097639.932579:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097640.444502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097640.764747:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417150976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097640.766115:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097640.766121:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097640.766124:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097640.766132:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2144 00000100:00080000:3.0:1766097640.766135:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097640.766140:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097640.766149:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097640.766163:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417151104/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097640.766218:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151104:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097640.766312:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097640.766671:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097640.766675:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097640.770776:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417150976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097640.773815:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097640.773825:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097640.773833:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151104:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097640.774105:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151040:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097640.957890:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097641.469023:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097641.980546:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097642.494594:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097643.004614:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097643.516483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097644.030518:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097644.541124:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097645.055701:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097645.566158:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097645.884663:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097645.884670:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097645.884674:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097645.884683:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2149 00000100:00080000:2.0:1766097645.884687:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097645.884692:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097645.884703:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097645.884718:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417151296/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097645.884783:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151296:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097645.884907:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097645.884945:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097645.884951:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417151168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097645.885139:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097645.885143:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097645.891750:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417151168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097645.891778:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097645.892219:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097645.892225:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097645.892230:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151296:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097646.084829:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097646.588515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097647.100679:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097647.612435:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097648.125126:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097648.636443:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097649.148649:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097649.660773:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097650.173281:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097650.684420:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097651.009654:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097651.009661:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097651.009665:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097651.009676:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2154 00000100:00080000:0.0:1766097651.009680:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1766097651.009685:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417151360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097651.009687:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097651.009699:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097651.009731:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ffe0f9f x1851885417151488/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097651.009760:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097651.009763:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097651.009807:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417151424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097651.010682:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151488:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097651.034636:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097651.034647:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097651.034657:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151488:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097651.034748:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417151360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097651.036138:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417151424:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097651.196998:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097651.708605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097652.220565:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097652.732602:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097653.244700:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097653.758735:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097654.268747:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097654.780456:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097655.292542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097655.804471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097656.125842:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097656.125850:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097656.125854:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097656.125864:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2160 00000100:00080000:3.0:1766097656.125867:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097656.125872:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097656.125882:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097656.125896:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417151680/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097656.125921:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097656.125925:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097656.125952:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097656.125968:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151680:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097656.126040:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097656.126046:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417151552:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097656.129556:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097656.133509:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417151552:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097656.133633:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097656.133638:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097656.133643:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151680:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097656.318187:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097656.828463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097657.340571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097657.852487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097658.365150:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097658.876472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097659.389487:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097659.900468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097660.413236:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097660.924475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097661.244566:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097661.244573:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097661.244577:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097661.244587:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2165 00000100:00080000:2.0:1766097661.244603:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097661.244608:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097661.244617:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097661.244631:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417151872/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097661.244653:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097661.244656:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097661.244721:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097661.244852:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151872:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097661.244905:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097661.244911:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417151744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097661.251910:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417151744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097661.251930:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417151808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097661.253288:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097661.253298:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097661.253306:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417151872:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097661.436627:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097661.948520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097662.460585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097662.972472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097663.484575:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097663.996472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097664.508929:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097665.020452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097665.534435:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097666.048833:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097666.364565:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097666.364573:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097666.364576:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097666.364586:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2170 00000100:00080000:3.0:1766097666.364590:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097666.364596:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097666.364605:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097666.364620:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417152064/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097666.364647:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097666.364651:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097666.364690:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417152000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097666.364809:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417151936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097666.364998:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097666.369261:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417152000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097666.371325:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417151936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097666.371445:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097666.371450:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097666.371454:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152064:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766097666.556915:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097667.068474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097667.580589:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097668.092480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097668.605419:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097669.119498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097669.629536:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097670.140875:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097670.653001:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097671.164649:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097671.485488:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417152128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097671.486596:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097671.486601:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097671.486614:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097671.486621:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2175 00000100:00080000:0.0:1766097671.486624:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097671.486629:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097671.486639:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097671.486655:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417152256/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097671.486700:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152256:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766097671.487267:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097671.487271:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097671.492334:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417152192:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097671.492541:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417152128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097671.498104:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097671.498114:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097671.498125:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152256:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097671.500191:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417152192:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097671.678403:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097672.188484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097672.700656:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097673.214751:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097673.726479:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097674.238793:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097674.748847:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097675.262859:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097675.772678:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097676.284598:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097676.606575:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097676.606583:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097676.606587:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097676.606597:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2180 00000100:00080000:2.0:1766097676.606600:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097676.606607:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097676.606618:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097676.606634:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b951c02b x1851885417152448/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097676.606658:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097676.606661:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097676.606887:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097676.607228:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417152320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097676.607278:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097676.607284:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417152384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097676.610585:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097676.610597:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097676.610605:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097676.611849:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417152320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097676.617583:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417152384:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097676.801190:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097677.308501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097677.826187:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097678.335417:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097678.844627:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097679.356472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097679.868776:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097680.380472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097680.892574:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097681.404478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097681.724573:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097681.724579:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097681.724582:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097681.724591:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2185 00000100:00080000:1.0:1766097681.724606:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097681.724611:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097681.724619:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097681.724632:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417152640/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097681.724654:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097681.724657:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097681.724679:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417152576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097681.724704:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152640:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097681.724783:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417152512:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097681.734415:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097681.734435:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097681.734445:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152640:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097681.741992:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417152512:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097681.746976:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417152576:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097681.916628:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097682.435180:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097682.940552:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097683.456781:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097683.964625:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097684.476439:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097684.988570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097685.500478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097686.012567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097686.524470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097686.844594:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097686.844601:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097686.844619:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097686.844629:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2190 00000100:00080000:3.0:1766097686.844632:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097686.844638:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097686.844648:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097686.844665:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbfb1ac7 x1851885417152832/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097686.844688:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097686.844692:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097686.844718:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417152768:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097686.844724:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152832:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097686.844809:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097686.844818:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417152704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097686.858690:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417152768:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097686.865658:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417152704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097686.867464:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097686.867474:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097686.867482:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417152832:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097687.036543:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097687.548462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097688.060534:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097688.572480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097689.084553:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097689.596613:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097690.108558:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097690.620510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097691.134481:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097691.644474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097691.964520:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097691.964528:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097691.964531:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097691.964541:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2195 00000100:00080000:3.0:1766097691.964544:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097691.964550:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097691.964560:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097691.964592:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417153024/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097691.964612:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097691.964616:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097691.964639:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417152960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097691.964707:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097691.964712:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417152896:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097691.964788:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153024:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097691.972063:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417152896:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097691.975846:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097691.975857:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097691.975865:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153024:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097691.976188:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417152960:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097692.156581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097692.668490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097693.180523:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097693.693291:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097694.212444:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097694.719313:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097695.228588:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097695.740480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097696.252572:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097696.766154:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097697.084660:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417153088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097697.086343:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417153152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097697.087058:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097697.087064:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097697.087068:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097697.087073:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2201 00000100:00080000:0.0:1766097697.087076:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097697.087081:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097697.087090:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097697.087109:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417153216/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097697.087175:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153216:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766097697.087623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097697.088210:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097697.089414:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417153088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097697.090591:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417153152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097697.090650:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097697.090654:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097697.090743:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153216:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097697.277595:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097697.788474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097698.301931:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097698.812476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097699.324578:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097699.836477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097700.348549:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097700.862427:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097701.374399:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097701.888192:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097702.205172:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097702.205179:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097702.205183:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097702.205193:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2206 00000100:00080000:3.0:1766097702.205196:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097702.205200:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097702.205210:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097702.205224:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417153408/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097702.205256:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097702.205260:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097702.205287:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417153344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097702.205393:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097702.205399:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417153280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097702.206038:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153408:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097702.211895:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417153280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097702.211946:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417153344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097702.218649:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097702.218658:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097702.218667:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153408:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097702.398246:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097702.908465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097703.428271:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097703.932733:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097704.445092:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097704.956490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097705.470453:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097705.980514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097706.492670:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097707.004486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097707.324664:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097707.324673:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097707.324676:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097707.324687:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2211 00000100:00080000:3.0:1766097707.324691:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097707.324697:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766097707.324706:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417153472:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1766097707.324708:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097707.324732:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417153600/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097707.324754:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097707.324758:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097707.324825:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097707.324834:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417153536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097707.324907:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153600:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097707.328052:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417153536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097707.328072:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417153472:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097707.331771:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097707.331785:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097707.331795:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153600:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097707.518283:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097708.028473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097708.541162:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097709.052727:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097709.564579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097710.078979:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097710.590944:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097711.100461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097711.613398:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097712.124471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097712.444522:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097712.444539:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097712.444554:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097712.444564:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2216 00000100:00080000:3.0:1766097712.444568:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097712.444573:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097712.444582:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097712.444597:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417153792/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097712.444618:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097712.444622:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097712.444701:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417153728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097712.444741:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153792:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097712.444794:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097712.444804:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417153664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:0.0:1766097712.451314:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:3.0:1766097712.452044:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417153664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097712.456870:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097712.456882:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097712.456894:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153792:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097712.460432:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417153728:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097712.636597:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097713.148469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097713.660574:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097714.172484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097714.684877:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097715.196474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097715.708576:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097716.222163:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097716.736110:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097717.246548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097717.564573:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417153856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097717.569002:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417153856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097717.569498:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417153920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097717.570671:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097717.570679:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097717.570682:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097717.570689:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2221 00000100:00080000:1.0:1766097717.570693:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097717.570699:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097717.571457:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097717.571490:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004229f3d5 x1851885417153984/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097717.571518:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097717.571521:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097717.571700:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153984:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097717.573168:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417153920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097717.575779:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097717.575789:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097717.575827:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417153984:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097717.756925:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097718.268736:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097718.781002:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097719.292531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097719.807921:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097720.316574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097720.829329:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097721.344816:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097721.852559:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097722.364487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097722.684627:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097722.684634:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097722.684637:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097722.684647:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2226 00000100:00080000:3.0:1766097722.684651:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097722.684656:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097722.684667:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097722.684682:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52cb2e2 x1851885417154176/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097722.684706:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097722.684709:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097722.684774:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154176:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097722.685595:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417154048:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097722.685649:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097722.685656:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417154112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097722.691970:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097722.691981:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097722.691990:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154176:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097722.692996:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417154048:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097722.693042:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417154112:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097722.877309:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097723.388477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097723.900789:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097724.412890:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097724.924573:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097725.436776:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097725.948559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097726.460477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097726.975814:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097727.484544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097727.804676:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097727.804683:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097727.804686:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097727.804696:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2231 00000100:00080000:0.0:1766097727.804699:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097727.804703:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097727.804712:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097727.804751:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000968f8523 x1851885417154368/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097727.804766:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097727.804769:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097727.804790:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417154304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097727.804867:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097727.804872:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417154240:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097727.804934:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154368:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766097727.808335:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097727.808346:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097727.808382:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154368:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097727.808911:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417154240:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097727.811306:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417154304:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097727.998034:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097728.508466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097729.020575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097729.532481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097730.048517:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097730.557232:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097731.068567:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097731.580472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097732.092595:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097732.604493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097732.924590:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097732.924613:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097732.924616:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097732.924626:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2236 00000100:00080000:3.0:1766097732.924629:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097732.924635:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097732.924643:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097732.924658:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417154560/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097732.924710:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097732.924714:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097732.924783:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154560:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097732.924935:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417154432:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097732.924972:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097732.924978:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417154496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097732.928386:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417154496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097732.930269:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417154432:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097732.931924:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097732.931938:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097732.931947:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154560:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097733.118484:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097733.628725:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097734.142991:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097734.652513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097735.165246:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097735.676600:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097736.192846:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097736.700610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097737.212571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097737.726874:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097738.044748:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417154624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097738.045731:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097738.045743:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417154688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097738.046809:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097738.046816:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097738.046820:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097738.046826:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2242 00000100:00080000:3.0:1766097738.046829:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097738.046833:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097738.046844:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097738.046867:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52cb2e2 x1851885417154752/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097738.046890:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097738.046893:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097738.046980:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154752:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097738.054625:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417154688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097738.054653:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417154624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097738.054898:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097738.054904:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097738.054908:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154752:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097738.236612:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097738.748470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097739.260670:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097739.772478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097740.284579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097740.796470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097741.308895:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097741.820472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097742.333097:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097742.844472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097743.164543:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097743.164550:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097743.164554:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097743.164563:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2247 00000100:00080000:3.0:1766097743.164567:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097743.164572:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097743.164582:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097743.164597:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417154944/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097743.164643:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097743.164646:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097743.164674:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417154880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097743.164682:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154944:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097743.165195:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097743.165204:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417154816:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097743.170559:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417154816:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097743.170583:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417154880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097743.171762:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097743.171772:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097743.171779:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417154944:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097743.356593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097743.868472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097744.380566:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097744.894958:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097745.408390:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097745.916471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097746.429258:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097746.940461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097747.453103:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097747.965966:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097748.284614:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155008:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097748.292647:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417155072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097748.292665:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097748.292671:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097748.292675:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097748.292681:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2252 00000100:00080000:0.0:1766097748.292684:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097748.292690:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097748.292715:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097748.292733:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417155136/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097748.292747:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097748.292750:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097748.292947:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155136:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097748.296652:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155008:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097748.300927:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417155072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097748.301050:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097748.301065:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097748.301069:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155136:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097748.476592:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097748.994028:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097749.504484:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097750.012619:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097750.524987:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097751.036484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097751.549169:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097752.060476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097752.574437:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097753.084545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097753.404596:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097753.404609:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097753.404612:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097753.404622:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2257 00000100:00080000:0.0:1766097753.404625:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097753.404634:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097753.404644:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097753.404664:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417155328/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097753.404693:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097753.404696:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097753.404797:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097753.404913:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097753.404921:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417155264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097753.405296:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155328:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097753.414068:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097753.414084:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097753.414098:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155328:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097753.414445:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417155264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097753.414462:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155200:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097753.598303:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097754.108673:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097754.620579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097755.132708:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097755.647852:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097756.156530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097756.670628:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097757.182585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097757.693863:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097758.204549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097758.524976:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097758.524984:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097758.524988:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097758.525001:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2262 00000100:00080000:2.0:1766097758.525005:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097758.525011:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097758.525044:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097758.525062:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000db8badc0 x1851885417155520/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097758.525088:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097758.525091:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097758.525497:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155520:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097758.548440:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417155392:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097758.548571:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097758.548578:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417155456:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097758.549162:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097758.549169:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097758.549175:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155520:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097758.559465:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417155456:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097758.559498:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417155392:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097758.717821:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097759.230219:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097759.743756:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097760.252521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097760.764673:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097761.276750:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097761.800901:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097762.300540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097762.812635:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097763.325079:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097763.644611:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097763.644620:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097763.644625:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097763.644639:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2267 00000100:00080000:2.0:1766097763.644643:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097763.644649:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097763.644661:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097763.644678:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417155712/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097763.644706:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097763.644709:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097763.644757:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417155584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097763.644853:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097763.644858:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417155648:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097763.645062:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155712:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097763.650954:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417155584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097763.651450:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417155648:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097763.654120:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097763.654131:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097763.654140:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155712:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097763.836872:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097764.349309:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097764.860573:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097765.378673:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097765.884758:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097766.396504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097766.908583:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097767.420493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097767.932557:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097768.444769:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097768.764588:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097768.764595:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097768.764598:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097768.764609:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2272 00000100:00080000:0.0:1766097768.764612:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097768.764617:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097768.764627:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097768.764644:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417155904/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097768.764667:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097768.764670:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097768.764750:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097768.764833:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097768.764839:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417155840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097768.764941:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155904:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766097768.773273:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097768.773285:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097768.773294:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417155904:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097768.774584:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417155840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097768.776207:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155776:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097768.956534:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097769.468488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097769.980593:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097770.493502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097771.005808:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097771.523574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097772.034237:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097772.544608:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097773.052603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097773.564491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097773.885714:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097773.890564:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417156032:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097773.891677:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097773.891685:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097773.891688:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097773.891700:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2277 00000100:00080000:0.0:1766097773.891703:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097773.891708:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097773.891718:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097773.891740:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52a4842 x1851885417156096/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097773.891761:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097773.891776:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097773.892710:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156096:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097773.900319:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417156032:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097773.900525:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097773.900530:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097773.900534:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156096:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097773.900764:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417155968:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097774.078732:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097774.588475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097775.100542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097775.612488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097776.125758:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097776.636643:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097777.150189:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097777.660479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097778.172573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097778.684519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097779.004774:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097779.004781:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097779.004785:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097779.004795:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2282 00000100:00080000:0.0:1766097779.004798:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097779.004804:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097779.004813:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097779.004827:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417156288/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097779.004849:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097779.004853:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097779.004874:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417156224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097779.004956:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097779.004962:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417156160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097779.005079:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097779.012241:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417156160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097779.012263:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417156224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097779.016979:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097779.016990:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097779.016999:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156288:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097779.197960:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097779.708463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097780.220610:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097780.735284:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097781.245158:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097781.756448:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097782.270229:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097782.780473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097783.292571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097783.804431:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097784.124585:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097784.124592:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097784.124596:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097784.124606:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2288 00000100:00080000:2.0:1766097784.124609:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097784.124615:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097784.124624:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097784.124637:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417156480/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097784.124680:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097784.124684:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097784.124709:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417156416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097784.124741:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156480:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097784.124798:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097784.124806:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417156352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097784.132043:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417156416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097784.132564:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097784.132570:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097784.132575:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156480:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097784.132718:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417156352:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097784.320563:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097784.828469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097785.340845:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097785.852982:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097786.364666:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097786.876481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097787.388969:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097787.900475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097788.413153:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097788.924479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097789.245091:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097789.245098:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097789.245101:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097789.245111:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2293 00000100:00080000:3.0:1766097789.245114:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097789.245120:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097789.245130:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097789.245161:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417156672/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097789.245185:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097789.245189:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097789.245215:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417156608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097789.245232:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156672:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097789.245321:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417156544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097789.251113:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417156544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097789.251384:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097789.251389:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097789.251394:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156672:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097789.253571:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417156608:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097789.436651:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097789.948490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097790.460602:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097790.972507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097791.485968:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097791.997223:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097792.508469:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097793.022433:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097793.533175:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097794.044469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097794.364523:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417156736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097794.369152:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097794.369159:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097794.369162:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097794.369172:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2298 00000100:00080000:3.0:1766097794.369175:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097794.369180:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097794.369189:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097794.369203:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417156864/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097794.369233:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097794.369237:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097794.369263:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417156800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097794.369281:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097794.370047:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417156736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097794.373072:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417156800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766097794.373639:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097794.373644:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097794.373650:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417156864:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097794.558653:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097795.068473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097795.580705:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097796.093037:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097796.605144:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097797.117255:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097797.630548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097798.141688:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097798.654387:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097799.164801:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097799.485286:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097799.485293:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097799.485297:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097799.485307:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2303 00000100:00080000:0.0:1766097799.485310:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097799.485315:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766097799.485318:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417156992:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1766097799.485325:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097799.485339:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417157056/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097799.485388:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097799.485391:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097799.485416:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157056:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097799.485426:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417156928:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097799.488903:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097799.488914:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097799.488922:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157056:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097799.489073:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417156928:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097799.489151:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417156992:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097799.679374:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097800.188473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097800.700610:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097801.212485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097801.724603:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097802.237958:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097802.749339:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097803.260470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097803.772513:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097804.284582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097804.604735:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097804.604742:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097804.604745:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097804.604755:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2308 00000100:00080000:3.0:1766097804.604757:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097804.604763:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097804.604772:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097804.604785:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417157248/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097804.604811:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097804.604815:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097804.604846:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097804.604883:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417157120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097804.604949:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157248:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097804.610949:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417157120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097804.610998:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097804.611091:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097804.611095:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097804.611099:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157248:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766097804.796662:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097805.311225:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097805.820542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097806.332475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097806.845384:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097807.356521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097807.868810:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097808.380487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097808.892526:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097809.404512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097809.724503:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097809.724511:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097809.724514:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097809.724524:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2313 00000100:00080000:3.0:1766097809.724527:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097809.724533:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097809.724543:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097809.724576:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbfb1ac7 x1851885417157440/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097809.724599:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097809.724602:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097809.724626:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097809.724663:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157440:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097809.724711:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097809.724722:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417157312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097809.733465:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097809.734573:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097809.734581:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097809.734589:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157440:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766097809.734731:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417157312:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097809.918767:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097810.428527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097810.940619:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097811.453135:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097811.965636:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097812.476638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097812.988971:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097813.500581:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097814.016172:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097814.524696:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097814.845250:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097814.845257:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097814.845261:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097814.845272:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2318 00000100:00080000:3.0:1766097814.845275:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097814.845280:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097814.845290:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097814.845308:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417157632/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097814.845332:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097814.845335:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097814.845374:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097814.845388:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097814.846608:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417157504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097814.849601:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417157504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097814.849695:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097814.849700:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097814.849704:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097814.858391:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157568:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766097815.036662:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097815.548474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097816.060923:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097816.575596:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097817.084560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097817.596481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097818.108582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097818.621982:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097819.133065:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097819.644433:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097819.964557:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097819.964564:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097819.964567:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097819.964577:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2323 00000100:00080000:3.0:1766097819.964580:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097819.964585:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097819.964594:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097819.964607:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417157824/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097819.964628:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097819.964632:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097819.964665:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157824:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097819.964665:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097819.964668:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417157696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097819.967972:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417157696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097819.968470:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417157760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097819.971673:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097819.971687:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097819.971698:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417157824:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097820.157684:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097820.668474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097821.180557:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097821.692475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097822.204522:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097822.718990:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097823.228583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097823.743176:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097824.254065:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097824.764477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097825.086105:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417157888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097825.086924:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097825.086936:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417157952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097825.087811:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097825.087820:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097825.087824:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097825.087832:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2329 00000100:00080000:0.0:1766097825.087836:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097825.087842:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097825.087880:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097825.087908:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005ed5ddd4 x1851885417158016/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097825.087937:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097825.087940:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097825.089734:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158016:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097825.098823:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417157888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097825.103441:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097825.103457:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097825.103471:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158016:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097825.103717:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417157952:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766097825.277480:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097825.788465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097826.301209:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097826.813060:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097827.324673:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097827.836519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097828.350846:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097828.860574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097829.372688:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097829.884702:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097830.207455:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097830.207464:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097830.207469:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097830.207482:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2334 00000100:00080000:2.0:1766097830.207486:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097830.207492:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097830.207506:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097830.207526:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1f2cb20 x1851885417158208/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097830.207597:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097830.207874:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097830.207919:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097830.207925:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097830.208536:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097830.208542:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766097830.214803:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097830.214818:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097830.214831:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097830.215486:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097830.222247:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158144:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097830.396707:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097830.908528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097831.421880:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097831.932557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097832.445298:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097832.956533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097833.473423:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097833.980472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097834.492552:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097835.004616:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097835.324608:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097835.324618:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097835.324623:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097835.324637:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2339 00000100:00080000:2.0:1766097835.324642:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097835.324649:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097835.324662:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097835.324678:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417158400/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097835.324704:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097835.324726:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097835.324778:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158400:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097835.324822:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097835.324947:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097835.324955:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097835.335168:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097835.335183:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097835.335196:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158400:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097835.335450:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097835.335464:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158272:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097835.518600:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097836.028531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097836.542566:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097837.053050:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097837.569298:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097838.077591:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097838.589346:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097839.100531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097839.612479:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097840.124470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097840.445099:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097840.445106:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097840.445109:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097840.445118:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2344 00000100:00080000:3.0:1766097840.445121:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097840.445126:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097840.445135:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097840.445147:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417158592/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097840.445168:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097840.445171:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097840.445193:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417158528:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097840.445209:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158592:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097840.445300:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097840.452661:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097840.452671:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417158528:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097840.452707:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097840.452713:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097840.452719:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158592:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766097840.636818:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097841.149951:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097841.667908:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097842.172466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097842.684566:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097843.196705:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097843.709238:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097844.220922:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097844.732570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097845.244599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097845.565088:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097845.565094:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097845.565098:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097845.565107:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2349 00000100:00080000:2.0:1766097845.565110:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097845.565116:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097845.565125:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097845.565137:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417158784/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097845.565157:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097845.565161:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097845.565238:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097845.566121:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097845.566929:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097845.566937:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097845.579690:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097845.579700:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097845.579710:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097845.582097:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097845.582116:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158656:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097845.760216:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097846.268461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097846.782882:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097847.292482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097847.805081:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097848.316480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097848.828908:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097849.340474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097849.852854:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097850.364699:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097850.684552:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097850.684558:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097850.684561:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097850.684571:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2354 00000100:00080000:2.0:1766097850.684574:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097850.684584:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097850.684593:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097850.684608:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417158976/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097850.684623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097850.684625:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097850.684668:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158976:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097850.684757:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097850.684789:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097850.684794:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097850.692700:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097850.692710:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097850.692720:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417158976:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097850.692894:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417158912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097850.692908:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417158848:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097850.877217:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097851.388470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097851.904745:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097852.412476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097852.926252:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097853.439054:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097853.948615:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097854.488086:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097854.973285:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097855.550436:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097855.804640:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097855.804648:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097855.804663:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097855.804672:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2359 00000100:00080000:1.0:1766097855.804676:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097855.804681:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097855.804690:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097855.804705:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000094c69f2b x1851885417159168/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097855.804728:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097855.804731:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097855.804755:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417159104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097855.804880:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097855.804886:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417159040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097855.804966:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159168:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097855.808234:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417159040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097855.808258:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417159104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766097855.808399:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097855.808404:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097855.808408:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159168:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097855.996752:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097856.572477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097857.020568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097857.597194:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097858.044554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097858.620466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097859.069238:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097859.644473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097860.093203:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097860.668562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097860.924593:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097860.924601:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097860.924604:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097860.924613:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2364 00000100:00080000:3.0:1766097860.924616:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097860.924620:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097860.924628:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097860.924642:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417159360/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097860.924663:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097860.924678:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097860.924701:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417159296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097860.924805:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097860.924810:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417159232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097860.924997:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097860.934192:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417159232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097860.936533:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417159296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097860.941716:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097860.941726:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097860.941757:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159360:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766097861.117059:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097861.692630:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097862.140560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097862.717191:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097863.167308:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097863.740483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097864.205959:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097864.764473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097865.212683:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097865.788532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097866.044651:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097866.044659:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097866.044663:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097866.044673:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2370 00000100:00080000:2.0:1766097866.044677:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097866.044682:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097866.044701:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097866.044716:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1d41706 x1851885417159552/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097866.044759:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097866.044763:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097866.044790:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417159488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097866.044882:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097866.044887:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417159424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097866.044972:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159552:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097866.052980:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417159488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1766097866.056382:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:2.0:1766097866.059341:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417159424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097866.072550:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097866.072562:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097866.072574:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159552:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097866.238572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097866.812461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097867.260801:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097867.836838:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097868.284550:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097868.860493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097869.308555:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097869.884477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097870.333102:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097870.909796:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097871.164561:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097871.164569:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097871.164572:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097871.164583:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2375 00000100:00080000:1.0:1766097871.164586:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097871.164592:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097871.164603:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097871.164617:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417159744/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097871.164644:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097871.164648:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097871.164704:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417159680:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097871.164793:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097871.164800:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417159616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097871.164902:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159744:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097871.168922:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097871.168933:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097871.168943:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159744:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097871.170899:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417159616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097871.170922:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417159680:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097871.356628:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097871.932464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097872.380981:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097872.956470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097873.404489:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097873.981199:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097874.428540:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097875.004482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097875.453140:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097876.028507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097876.288178:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097876.288186:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097876.288189:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097876.288199:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2380 00000100:00080000:0.0:1766097876.288203:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097876.288208:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097876.288218:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097876.288233:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000512a78fc x1851885417159936/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097876.288278:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097876.288281:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097876.288377:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417159872:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097876.288472:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097876.288479:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417159808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097876.288703:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159936:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097876.292446:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417159808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097876.292471:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417159872:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097876.293091:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097876.293096:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097876.293101:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417159936:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766097876.476560:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097877.052462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097877.501979:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097878.076476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097878.524551:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097879.100781:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097879.548723:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097880.124484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097880.572490:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097881.148483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097881.406946:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097881.406953:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097881.406956:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097881.406966:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2385 00000100:00080000:1.0:1766097881.406970:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097881.406975:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097881.406984:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097881.406997:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417160128/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097881.407018:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097881.407022:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097881.407047:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417160064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097881.407166:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097881.407173:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417160000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097881.407267:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160128:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766097881.414695:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766097881.414706:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766097881.414727:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160128:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097881.416061:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417160064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097881.416730:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417160000:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097881.596661:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097882.173655:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097882.622388:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097883.196687:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097883.644489:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097884.220640:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097884.668602:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097885.244465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097885.692558:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097886.269400:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097886.524587:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097886.524595:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097886.524599:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097886.524610:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2390 00000100:00080000:1.0:1766097886.524613:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097886.524619:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097886.524629:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097886.524645:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9051765 x1851885417160320/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097886.524663:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097886.524667:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097886.524694:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417160256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097886.524784:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097886.524791:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417160192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097886.526392:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160320:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097886.529437:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417160192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097886.529469:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417160256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097886.530277:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097886.530284:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097886.530289:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160320:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097886.716725:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097887.293376:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097887.742709:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097888.316894:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097888.764564:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097889.340464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097889.790653:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097890.364476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097890.813144:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097891.388472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097891.645547:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097891.645554:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097891.645558:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097891.645569:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2395 00000100:00080000:0.0:1766097891.645573:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097891.645594:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097891.645604:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097891.645621:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417160512/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097891.645647:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097891.645651:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097891.645680:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417160448:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097891.645772:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766097891.645779:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417160384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097891.646520:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160512:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766097891.650460:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097891.650471:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097891.650480:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160512:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766097891.650579:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417160384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097891.650595:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417160448:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766097891.840180:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097892.413540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097892.861426:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097893.436724:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097893.884622:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097894.460478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097894.908551:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097895.484532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097895.933211:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097896.509390:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097896.764611:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097896.764618:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097896.764622:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097896.764631:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2400 00000100:00080000:1.0:1766097896.764634:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097896.764640:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097896.764650:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097896.764664:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417160704/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097896.764686:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097896.764690:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097896.764728:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160704:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097896.764753:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417160576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097896.764823:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097896.764832:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417160640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097896.770111:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097896.770123:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097896.770133:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160704:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097896.773280:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417160576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097896.773574:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417160640:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097896.956631:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097897.533176:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097897.985295:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097898.556498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097899.004584:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097899.580549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097900.028579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097900.605297:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097901.052757:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097901.628447:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097901.884604:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097901.884611:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097901.884614:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097901.884624:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2405 00000100:00080000:2.0:1766097901.884642:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097901.884648:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097901.884661:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097901.884661:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417160768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097901.884681:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009119b13e x1851885417160896/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097901.884720:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097901.884724:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097901.884747:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417160832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097901.884866:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097901.888414:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417160832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097901.889225:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417160768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097901.890762:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097901.890772:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097901.890779:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417160896:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097902.077204:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097902.652524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097903.100619:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097903.676507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097904.124726:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097904.705964:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097905.149333:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097905.724473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097906.172716:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097906.748712:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097907.004697:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097907.004704:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097907.004708:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097907.004717:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2410 00000100:00080000:2.0:1766097907.004720:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097907.004725:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097907.004734:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097907.004735:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417160960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097907.004749:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417161088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097907.004772:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097907.004776:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097907.004799:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417161024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097907.004820:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161088:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097907.006773:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417161024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097907.006860:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417160960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097907.008674:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097907.008683:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097907.008688:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161088:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766097907.196579:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097907.772514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097908.222589:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097908.796474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097909.244562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097909.821441:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097910.269899:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097910.844487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097911.293498:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766097911.868607:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097912.124554:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097912.124561:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097912.124564:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097912.124573:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2416 00000100:00080000:3.0:1766097912.124577:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097912.124583:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097912.124596:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097912.124612:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417161280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097912.124655:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097912.124659:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097912.124739:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161280:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097912.125067:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417161216:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097912.125104:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097912.125110:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417161152:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097912.128327:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097912.128339:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097912.128414:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161280:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097912.128621:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417161152:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097912.128638:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417161216:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097912.317652:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097912.892524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097913.342268:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097913.917219:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097914.364651:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097914.940996:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097915.388620:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097915.964638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097916.414599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097916.988731:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097917.244839:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097917.244847:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097917.244850:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097917.244860:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2421 00000100:00080000:1.0:1766097917.244863:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097917.244869:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097917.244879:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097917.244893:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417161472/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097917.244917:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417161344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097917.244919:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097917.244923:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097917.244959:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417161408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097917.245021:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161472:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766097917.251879:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417161344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097917.251991:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417161408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097917.260750:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097917.260762:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097917.260772:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161472:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097917.437012:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097918.012564:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097918.461811:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097919.045009:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097919.484564:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097920.060752:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097920.509536:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097921.084481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097921.532687:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097922.108538:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766097922.364534:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097922.364541:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097922.364545:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097922.364570:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2426 00000100:00080000:1.0:1766097922.364573:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097922.364579:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097922.364589:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097922.364607:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417161664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766097922.364628:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097922.364631:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097922.364657:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417161600:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097922.364679:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161664:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097922.364752:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097922.364761:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417161536:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766097922.369154:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417161536:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097922.369395:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097922.369426:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097922.369435:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161664:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766097922.373635:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417161600:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097922.556648:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097923.135396:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097923.581647:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097924.156466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097924.610127:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097925.183557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097925.629288:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097926.204478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097926.652544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097927.228485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766097927.484803:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417161728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097927.487204:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417161792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097927.499281:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097927.499291:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097927.499295:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766097927.499444:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417161792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097927.499504:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417161728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097927.499608:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2431 00000100:00080000:3.0:1766097927.499612:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097927.499656:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097927.499668:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097927.499693:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006796bf6d x1851885417161856/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097927.499759:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161856:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766097927.500315:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097927.500320:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766097927.502040:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097927.502049:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097927.502056:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417161856:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766097927.678010:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097928.253807:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097928.700556:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097929.276808:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097929.724788:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097930.300517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097930.749245:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097931.332503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097931.772548:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097932.348464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097932.604611:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097932.604619:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097932.604623:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097932.604633:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2436 00000100:00080000:0.0:1766097932.604636:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097932.604642:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097932.604651:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097932.604665:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000042205631 x1851885417162048/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097932.604690:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097932.604694:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097932.604737:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162048:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097932.604781:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417161920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097932.604827:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097932.604833:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417161984:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097932.608961:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097932.608970:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097932.608980:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162048:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097932.610040:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417161984:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097932.610051:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417161920:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097932.796609:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097933.372472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097933.820545:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097934.396465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097934.844621:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097935.420466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097935.870548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097936.444466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097936.894385:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097937.468497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097937.725128:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097937.725146:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097937.725150:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097937.725171:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2441 00000100:00100000:3.0:1766097937.725173:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417162112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097937.725175:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097937.725182:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097937.725197:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097937.725216:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417162240/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766097937.725267:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097937.725274:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417162176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097937.725529:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162240:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766097937.725710:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097937.725714:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097937.729253:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417162112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097937.729494:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097937.729499:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097937.729503:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162240:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097937.729621:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417162176:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766097937.916899:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097938.494465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097938.940535:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097939.516499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097939.967510:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097940.540472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097940.988573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097941.564473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097942.014046:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097942.588470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766097942.844565:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417162304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097942.845295:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766097942.845303:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417162368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097942.845709:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097942.845715:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097942.845718:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097942.845724:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2446 00000100:00080000:2.0:1766097942.845726:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097942.845731:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097942.845740:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097942.845771:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417162432/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097942.845793:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097942.845796:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097942.845825:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162432:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097942.849804:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417162368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097942.849808:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097942.849815:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097942.849821:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162432:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097942.849828:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417162304:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097943.036992:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097943.613138:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097944.061384:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097944.636472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097945.085136:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097945.661839:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097946.108513:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097946.684476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766097947.132537:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097947.708463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097947.965926:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097947.965934:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097947.965938:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097947.965947:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2451 00000100:00080000:3.0:1766097947.965950:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097947.965956:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097947.965966:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097947.965980:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417162624/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097947.966055:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162624:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097947.966185:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417162496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097947.966221:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097947.966225:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417162560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766097947.966662:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097947.966667:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766097947.975447:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097947.975458:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097947.975486:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162624:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766097947.975822:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417162560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766097947.980491:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417162496:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097948.156594:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097948.732473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097949.180573:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097949.756475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097950.205060:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097950.780508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097951.228912:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097951.810148:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097952.255139:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097952.828477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766097953.086181:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417162688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097953.087297:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766097953.087309:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417162752:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097953.089533:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097953.089553:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766097953.089559:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766097953.089567:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2457 00000100:00080000:1.0:1766097953.089571:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766097953.089579:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766097953.089592:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766097953.089616:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417162816/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766097953.089689:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162816:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766097953.089786:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766097953.089789:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097953.093664:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417162688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097953.093719:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417162752:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097953.095855:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097953.095871:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097953.095883:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417162816:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097953.277311:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097953.852452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097954.300668:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097954.876601:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097955.324661:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097955.900625:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097956.349490:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097956.926403:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097957.374298:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097957.948474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097958.204600:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097958.204608:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097958.204612:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097958.204624:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2462 00000100:00080000:2.0:1766097958.204628:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766097958.204634:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097958.204644:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097958.204664:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417163008/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097958.204695:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097958.204698:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097958.204754:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163008:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097958.205519:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417162944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097958.205564:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097958.205571:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417162880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097958.210858:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417162880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097958.210882:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417162944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097958.210948:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097958.210953:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097958.210957:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163008:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766097958.396794:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097958.972444:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097959.422539:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097959.996473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097960.445544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097961.020475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097961.469078:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097962.045154:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097962.493083:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097963.068484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766097963.324563:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097963.324569:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766097963.324573:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766097963.324583:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2467 00000100:00080000:2.0:1766097963.324586:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1766097963.324604:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417163072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766097963.324607:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766097963.324617:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766097963.324633:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417163200/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766097963.324657:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766097963.324661:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097963.324695:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163200:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097963.324704:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417163136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766097963.328619:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766097963.328631:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766097963.328640:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163200:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097963.328728:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417163136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097963.328756:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417163072:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097963.516600:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097964.092470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097964.542781:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097965.117245:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097965.569316:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097966.140477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097966.588583:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097967.164485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097967.613041:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097968.188459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097968.444834:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097968.444842:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097968.444845:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097968.444856:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2472 00000100:00080000:0.0:1766097968.444859:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097968.444865:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097968.444874:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097968.444889:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000030e2f50e x1851885417163392/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097968.444911:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097968.444915:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097968.444968:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417163328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097968.445849:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417163264:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766097968.445907:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163392:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097968.457485:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417163264:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097968.457587:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417163328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097968.457860:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097968.457865:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097968.457869:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163392:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097968.640101:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097969.212687:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097969.660633:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097970.236722:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097970.685305:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097971.260550:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097971.708526:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097972.284473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097972.734560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097973.308449:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766097973.564577:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417163456:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097973.565987:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417163520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766097973.566965:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097973.566973:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097973.566976:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097973.566982:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2477 00000100:00080000:0.0:1766097973.566986:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097973.566991:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097973.567001:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097973.567034:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2c84c91 x1851885417163584/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766097973.567092:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163584:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766097973.569329:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097973.569338:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766097973.569967:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417163520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766097973.570386:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417163456:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097973.571662:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097973.571670:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097973.571691:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163584:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766097973.756954:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097974.333504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097974.783210:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097975.356467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097975.805516:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097976.380524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097976.829172:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097977.405003:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097977.853762:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097978.428469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097978.685412:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097978.685419:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097978.685423:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097978.685435:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2482 00000100:00080000:0.0:1766097978.685439:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097978.685446:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097978.685456:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097978.685471:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417163776/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097978.685494:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097978.685497:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097978.685533:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163776:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097978.685590:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417163712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097978.685631:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766097978.685637:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417163648:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097978.693906:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097978.693916:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097978.693925:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163776:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097978.694329:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417163712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097978.694349:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417163648:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766097978.878437:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097979.452475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097979.901752:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097980.476490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097980.927476:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097981.500465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097981.948543:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097982.524557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097982.972570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097983.549113:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097983.806440:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097983.806447:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097983.806450:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097983.806461:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2487 00000100:00080000:3.0:1766097983.806464:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097983.806470:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097983.806480:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097983.806494:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417163968/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097983.806518:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097983.806521:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766097983.806551:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417163904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097983.806621:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766097983.806625:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417163840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766097983.806928:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163968:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097983.809400:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417163840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097983.809425:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417163904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097983.809518:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097983.809523:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097983.809527:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417163968:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766097983.996605:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097984.572477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097985.020676:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097985.596475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097986.046252:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097986.620789:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097987.068592:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097987.646307:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097988.092554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097988.668485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097988.924541:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097988.924549:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097988.924552:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097988.924563:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2492 00000100:00080000:3.0:1766097988.924566:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097988.924586:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097988.924596:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097988.924613:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004951ed42 x1851885417164160/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097988.924637:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097988.924641:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097988.924690:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164160:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097988.924773:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417164032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097988.924864:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766097988.924871:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417164096:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097988.931496:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097988.931506:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097988.931516:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164160:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097988.933775:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417164096:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097988.933808:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417164032:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097989.116714:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097989.692479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097990.140837:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097990.716510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097991.166734:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097991.742100:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097992.190961:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097992.764472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097993.213505:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097993.792032:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766097994.044497:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097994.044504:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766097994.044507:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766097994.044519:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2498 00000100:00080000:3.0:1766097994.044522:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766097994.044527:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766097994.044536:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766097994.044549:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbfb1ac7 x1851885417164352/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766097994.044565:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766097994.044569:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766097994.044656:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766097994.044942:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417164288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766097994.044990:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766097994.044997:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417164224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766097994.051656:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766097994.051672:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766097994.051681:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766097994.059020:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417164224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766097994.059044:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417164288:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766097994.239252:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097994.812468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097995.261124:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097995.836975:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766097996.286108:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097996.860476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097997.313102:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097997.884491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766097998.332519:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097998.908475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766097999.164665:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097999.164672:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766097999.164676:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766097999.164713:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2503 00000100:00080000:0.0:1766097999.164717:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766097999.164723:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766097999.164733:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766097999.164750:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417164544/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766097999.164776:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766097999.164779:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766097999.164853:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164544:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766097999.164990:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417164416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097999.165024:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766097999.165040:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417164480:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097999.167378:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417164480:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766097999.170469:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417164416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766097999.170891:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766097999.170896:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766097999.170901:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164544:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766097999.357422:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766097999.932491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098000.380719:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098000.959565:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098001.404539:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098001.980470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098002.430047:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098003.006331:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098003.452578:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098004.028530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098004.284624:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417164608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098004.284781:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098004.284789:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417164672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098004.285082:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098004.285094:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098004.285097:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098004.285102:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2508 00000100:00080000:2.0:1766098004.285105:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098004.285110:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098004.285120:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098004.285137:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1f2cb20 x1851885417164736/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098004.285189:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766098004.286349:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098004.286367:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098004.287912:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417164608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098004.290118:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417164672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098004.291883:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098004.291895:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098004.291904:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164736:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098004.478321:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098005.052656:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098005.502176:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098006.076560:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098006.524616:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098007.100592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098007.548599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098008.124530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098008.572667:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098009.149176:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098009.406513:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098009.406522:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098009.406527:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098009.406541:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2513 00000100:00080000:1.0:1766098009.406546:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098009.406553:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098009.406573:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098009.406593:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f7eb885 x1851885417164928/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098009.406617:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098009.406621:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098009.406663:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417164864:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098009.406788:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098009.406795:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417164800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098009.406900:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164928:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766098009.411836:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098009.411849:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098009.411861:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417164928:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098009.412857:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417164800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098009.412907:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417164864:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098009.598320:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098010.173116:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098010.624731:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098011.196841:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098011.646996:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098012.222269:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098012.668930:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098013.244518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098013.694085:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098014.268773:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098014.524695:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098014.524704:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098014.524709:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098014.524723:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2518 00000100:00080000:0.0:1766098014.524727:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098014.524733:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098014.524745:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098014.524764:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ffe0f9f x1851885417165120/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098014.524790:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098014.524793:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098014.524851:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165120:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098014.524875:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417164992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098014.524977:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098014.524983:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417165056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098014.534826:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417164992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098014.535764:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417165056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098014.536674:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098014.536680:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098014.536686:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165120:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766098014.718498:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098015.295344:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098015.741880:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098016.317335:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098016.765323:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098017.340857:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098017.790887:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098018.365950:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098018.812556:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098019.392265:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098019.644546:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098019.644729:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098019.644733:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098019.644745:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2523 00000100:00080000:3.0:1766098019.644748:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098019.644756:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098019.644767:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098019.644787:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417165312/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098019.644845:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165312:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766098019.644989:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098019.644992:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098019.645011:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417165248:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098019.645048:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098019.645053:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417165184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098019.649956:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417165184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098019.651511:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417165248:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1766098019.658213:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:0.0:1766098019.672566:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098019.672578:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098019.672590:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165312:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098019.839489:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098020.412469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098020.860764:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098021.437082:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098021.884680:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098022.460488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098022.908886:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098023.484473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098023.932559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098024.508480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098024.764587:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098024.764595:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098024.764598:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098024.764607:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2528 00000100:00080000:0.0:1766098024.764610:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098024.764615:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098024.764625:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098024.764641:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000968f8523 x1851885417165504/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098024.764666:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098024.764669:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098024.764701:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417165440:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098024.764703:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165504:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098024.764915:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417165376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098024.768941:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098024.768956:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098024.768967:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165504:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098024.769255:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417165376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098024.769531:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417165440:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098024.961548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098025.532600:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098025.981115:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098026.558235:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098027.006668:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098027.580483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098028.030697:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098028.609275:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098029.053022:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098029.628920:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098029.884671:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098029.884680:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098029.884685:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098029.884700:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2533 00000100:00080000:0.0:1766098029.884705:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098029.884713:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098029.884733:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098029.884741:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417165568:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098029.884758:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004229f3d5 x1851885417165696/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098029.884793:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098029.884797:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098029.884857:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098029.884867:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417165632:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098029.885175:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165696:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098029.889384:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098029.889399:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098029.889411:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165696:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098029.889631:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417165568:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098029.891007:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417165632:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766098030.077650:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098030.652513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098031.101450:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098031.676516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098032.124762:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098032.703033:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098033.148899:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098033.724503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098034.172969:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098034.748493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098035.005476:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098035.005482:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098035.005485:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098035.005494:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2538 00000100:00080000:2.0:1766098035.005497:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098035.005502:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098035.005510:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098035.005523:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417165888/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098035.005546:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098035.005549:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098035.005571:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417165824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098035.005646:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098035.005651:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417165760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098035.007953:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165888:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098035.008765:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417165760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098035.010786:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417165824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098035.012819:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098035.012832:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098035.012843:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417165888:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098035.197041:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098035.772658:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098036.220806:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098036.796504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098037.244592:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098037.820498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098038.269386:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098038.844491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098039.292683:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098039.868524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098040.124602:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098040.124610:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098040.124614:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098040.124625:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2544 00000100:00100000:0.0:1766098040.124626:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417166016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098040.124630:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098040.124651:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098040.124662:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098040.124684:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417166080/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098040.124710:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1766098040.124711:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1766098040.124716:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098040.124717:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417165952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098040.124805:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166080:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098040.127999:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417165952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098040.128029:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417166016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098040.128040:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098040.128045:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098040.128066:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166080:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098040.317124:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098040.893155:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098041.341064:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098041.917107:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098042.364958:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098042.940544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098043.388676:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098043.964756:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098044.413495:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098044.988507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098045.244605:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098045.244611:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098045.244615:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098045.244625:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2549 00000100:00080000:3.0:1766098045.244628:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 6s 00000100:00080000:3.0:1766098045.244634:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098045.244643:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098045.244656:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417166272/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098045.244678:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098045.244682:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098045.244722:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166272:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098045.244771:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417166144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098045.244880:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098045.244903:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417166208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098045.252995:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417166144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098045.261549:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098045.261563:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098045.261575:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166272:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098045.261811:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417166208:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098045.436685:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098046.012482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098046.460980:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098047.036482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098047.484958:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098048.060479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098048.508550:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098049.084467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098049.532583:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098050.108476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098050.364744:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417166336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098050.365039:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098050.365046:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417166400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098050.365196:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098050.365200:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098050.365203:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098050.365208:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2554 00000100:00080000:0.0:1766098050.365212:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:0.0:1766098050.365217:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098050.365226:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098050.365251:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417166464/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098050.365303:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166464:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766098050.365999:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098050.366005:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098050.369881:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417166336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098050.371538:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417166400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098050.371638:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098050.371643:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098050.371647:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166464:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098050.557566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098051.132465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098051.580729:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098052.156479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098052.605764:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098053.181749:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098053.628569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098054.204635:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098054.657101:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098055.230454:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098055.484543:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098055.484551:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098055.484554:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098055.484564:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2559 00000100:00080000:0.0:1766098055.484568:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:0.0:1766098055.484574:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098055.484583:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098055.484600:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417166656/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098055.484623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098055.484626:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098055.484651:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417166592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098055.484735:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098055.484769:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417166528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098055.486824:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166656:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098055.487772:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417166592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098055.489948:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417166528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098055.490061:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098055.490067:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098055.490072:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166656:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098055.679651:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098056.252711:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098056.701698:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098057.277325:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098057.724838:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098058.300568:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098058.749057:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098059.324693:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098059.772541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098060.348439:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098060.605425:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098060.605433:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098060.605437:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098060.605447:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2564 00000100:00080000:1.0:1766098060.605451:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098060.605458:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098060.605469:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098060.605489:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417166848/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098060.605512:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098060.605515:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098060.605595:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417166720:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098060.605687:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098060.605693:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417166784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098060.605881:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166848:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098060.613129:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098060.613144:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098060.613174:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417166848:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098060.613305:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417166784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098060.613755:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417166720:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766098060.796960:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098061.373588:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098061.820654:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098062.396934:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098062.844631:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098063.422986:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098063.869641:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098064.444473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098064.894070:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098065.469298:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098065.724536:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098065.724542:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098065.724545:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098065.724552:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2569 00000100:00080000:2.0:1766098065.724555:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766098065.724560:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098065.724570:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098065.724585:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417167040/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098065.724603:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098065.724606:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098065.724635:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417166976:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098065.724678:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167040:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098065.724717:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098065.724726:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417166912:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098065.729500:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417166912:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098065.731287:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417166976:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098065.731437:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098065.731443:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098065.731448:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167040:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098065.916687:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098066.496464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098066.940532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098067.516494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098067.964560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098068.540589:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098068.990649:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098069.308638:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417167104:192.168.203.152@tcp:17:kworker/1:2.0 00000100:00100000:0.0:1766098069.315825:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417167104:192.168.203.152@tcp:17:kworker/1:2.0 00000100:00100000:2.0:1766098069.564532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098070.017985:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098070.588476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098070.844600:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098070.844608:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098070.844612:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098070.844623:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2574 00000100:00080000:0.0:1766098070.844626:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098070.844633:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098070.844644:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098070.844647:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417167168:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098070.844664:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417167296/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098070.844685:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098070.844688:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098070.844720:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417167232:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098070.845041:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167296:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098070.847135:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417167168:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098070.854072:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417167232:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098070.855510:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098070.855516:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098070.855522:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167296:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098071.038349:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098071.612517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098072.060639:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098072.636497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098073.091389:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098073.660485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098074.111292:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098074.684568:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098075.132569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098075.710922:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098075.964622:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098075.964627:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098075.964631:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098075.964640:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2579 00000100:00080000:3.0:1766098075.964643:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098075.964648:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098075.964657:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098075.964670:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417167488/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098075.964690:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098075.964693:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098075.964715:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417167424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098075.964782:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098075.964787:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417167360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098075.964886:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167488:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098075.972497:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417167360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098075.972518:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417167424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098075.972741:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098075.972745:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098075.972750:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167488:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098076.162195:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098076.732482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098077.181854:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098077.757273:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098078.210338:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098078.780515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098079.228581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098079.804476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098080.253662:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098080.828589:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098081.095581:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098081.095587:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098081.095591:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098081.095601:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2585 00000100:00080000:2.0:1766098081.095604:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766098081.095610:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098081.095619:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098081.095633:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417167680/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098081.095654:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098081.095657:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098081.095772:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417167552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098081.095854:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098081.095859:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417167616:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098081.095938:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167680:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766098081.099138:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098081.099146:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098081.099154:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167680:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098081.100226:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417167552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098081.101128:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417167616:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098081.277747:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098081.852477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098082.301645:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098082.876472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098083.324609:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098083.900463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098084.351508:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098084.924631:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098085.372852:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098085.948474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098086.204591:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098086.204597:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098086.204601:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098086.204618:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2590 00000100:00080000:1.0:1766098086.204622:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098086.204628:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098086.204637:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098086.204650:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417167872/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098086.204719:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167872:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766098086.205783:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098086.205789:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098086.205979:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417167744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098086.206038:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098086.206044:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417167808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098086.228029:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417167808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098086.228050:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417167744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098086.228579:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098086.228584:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098086.228589:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417167872:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098086.396653:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098086.972463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098087.421546:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098087.996585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098088.445130:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098089.021940:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098089.468655:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098090.044468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098090.494849:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098091.068493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098091.325347:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098091.325387:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1766098091.325388:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417167936:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098091.325391:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098091.325396:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2595 00000100:00080000:3.0:1766098091.325398:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098091.325403:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098091.325411:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098091.325439:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006a1d607 x1851885417168064/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098091.325452:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098091.325455:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098091.325468:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098091.325474:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417168000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098091.325601:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168064:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098091.331527:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417168000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098091.331545:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417167936:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098091.335433:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098091.335443:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098091.335451:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168064:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098091.522094:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098092.092481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098092.540576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098093.116462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098093.564524:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098094.140475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098094.588588:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098095.165340:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098095.612570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098096.190340:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098096.444605:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098096.444613:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098096.444617:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098096.444628:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2600 00000100:00080000:1.0:1766098096.444632:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1766098096.444640:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417168128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098096.444640:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098096.444655:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098096.444684:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417168256/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098096.444718:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098096.444721:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098096.444746:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417168192:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098096.444772:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168256:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098096.450292:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098096.450304:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098096.450314:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168256:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098096.450424:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417168128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098096.450841:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417168192:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098096.637200:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098097.212484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098097.661132:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098098.236478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098098.684950:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098099.261700:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098099.709128:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098100.287222:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098100.733257:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098101.308487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098101.566972:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098101.566979:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098101.566983:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098101.566993:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2605 00000100:00080000:3.0:1766098101.566996:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098101.567001:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098101.567011:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098101.567027:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417168448/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098101.567051:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098101.567055:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098101.567094:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417168384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098101.567169:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098101.567174:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417168320:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098101.567240:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168448:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098101.570526:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417168384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098101.572807:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417168320:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098101.577835:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098101.577847:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098101.577856:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168448:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098101.756531:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098102.332474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098102.781239:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098103.356489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098103.804557:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098104.380478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098104.828567:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098105.404476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098105.853263:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098106.428485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098106.684545:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417168512:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098106.686032:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417168576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098106.686242:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098106.686246:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098106.686249:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098106.686254:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2610 00000100:00080000:3.0:1766098106.686256:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098106.686261:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098106.686270:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098106.686286:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417168640/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098106.686342:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168640:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766098106.687767:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098106.687774:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766098106.695041:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098106.695052:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098106.695060:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168640:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098106.695142:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417168576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098106.695676:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417168512:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098106.876768:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098107.452605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098107.900646:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098108.476474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098108.934302:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098109.500475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098109.948731:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098110.524497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098110.972546:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098111.548489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098111.804601:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098111.804608:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098111.804611:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098111.804621:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2615 00000100:00080000:1.0:1766098111.804624:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098111.804629:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098111.804638:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098111.804653:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417168832/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098111.804676:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098111.804680:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098111.804701:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417168768:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098111.804791:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098111.804797:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417168704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098111.804883:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168832:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098111.809648:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098111.809660:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098111.809670:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417168832:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098111.815186:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417168704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098111.815208:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417168768:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098111.999542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098112.572682:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098113.021079:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098113.596474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098114.046650:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098114.620471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098115.068882:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098115.644472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098116.092542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098116.668477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098116.928491:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098116.928498:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098116.928501:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098116.928511:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2620 00000100:00080000:1.0:1766098116.928514:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098116.928520:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098116.928529:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098116.928544:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417169024/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098116.928611:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169024:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098116.928903:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417168896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098116.928955:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098116.928961:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417168960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098116.929577:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098116.929582:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766098116.936576:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098116.936589:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098116.936598:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169024:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098116.949113:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417168960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098116.949128:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417168896:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098117.118153:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098117.692489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098118.140724:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098118.716527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098119.164734:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098119.740682:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098120.188909:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098120.764621:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098121.213418:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098121.788484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098122.045552:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098122.045560:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098122.045566:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098122.045581:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2626 00000100:00080000:3.0:1766098122.045586:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098122.045592:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098122.045605:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098122.045611:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417169088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098122.045630:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417169216/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098122.045656:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098122.045660:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098122.045726:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098122.045735:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417169152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098122.045952:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169216:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098122.050714:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417169152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098122.050731:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417169088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098122.056613:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098122.056623:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098122.056631:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169216:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098122.236577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098122.815807:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098123.264460:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098123.836480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098124.284592:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098124.860469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098125.309015:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098125.884510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098126.333296:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098126.908486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098127.164599:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417169280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098127.165632:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417169344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098127.165728:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098127.165733:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098127.165737:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098127.165743:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2631 00000100:00080000:1.0:1766098127.165746:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098127.165751:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098127.165773:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098127.165789:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417169408/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098127.165849:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169408:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766098127.166549:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098127.166554:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098127.170553:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417169280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098127.172287:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417169344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098127.174154:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098127.174163:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098127.174171:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169408:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098127.356582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098127.932479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098128.382922:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098128.956507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098129.408155:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098129.980478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098130.428536:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098131.004475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098131.453147:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098132.028547:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098132.286847:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098132.286854:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098132.286858:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098132.286869:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2636 00000100:00100000:1.0:1766098132.286871:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417169472:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098132.286873:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098132.286882:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098132.286893:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098132.286910:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000968f8523 x1851885417169600/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098132.286937:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098132.286941:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098132.286960:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417169536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098132.287051:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169600:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098132.293035:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417169536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098132.294126:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417169472:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098132.294276:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098132.294282:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098132.294286:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169600:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098132.476866:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098133.052576:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098133.500552:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098134.077957:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098134.526979:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098135.100521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098135.549232:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098136.124535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098136.572661:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098137.148578:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098137.406090:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098137.406097:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098137.406100:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098137.406109:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2641 00000100:00080000:1.0:1766098137.406113:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098137.406118:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098137.406127:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098137.406139:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417169792/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098137.406153:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098137.406156:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098137.406293:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417169664:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098137.406396:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098137.406402:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417169728:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098137.408207:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169792:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1766098137.414482:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098137.414493:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098137.414503:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169792:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098137.414609:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417169664:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098137.414635:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417169728:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098137.597098:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098138.173587:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098138.620877:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098139.197818:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098139.645131:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098140.220506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098140.670730:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098141.249946:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098141.692714:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098142.268898:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098142.524532:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098142.524538:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098142.524554:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098142.524564:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2646 00000100:00080000:1.0:1766098142.524566:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098142.524571:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098142.524581:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098142.524594:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417169984/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098142.524623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098142.524626:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098142.524659:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417169920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098142.524751:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098142.524758:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417169856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098142.524863:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169984:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098142.532009:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417169920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098142.536493:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098142.536504:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098142.536512:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417169984:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098142.536583:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417169856:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098142.716560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098143.292471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098143.740582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098144.316518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098144.764562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098145.343489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098145.790120:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098146.364475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098146.814251:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098147.389859:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098147.644584:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098147.644593:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098147.644596:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098147.644619:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2651 00000100:00080000:0.0:1766098147.644622:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098147.644628:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098147.644639:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098147.644655:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ffe0f9f x1851885417170176/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098147.644755:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170048:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098147.644847:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098147.644853:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417170112:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098147.644955:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098147.644959:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098147.645263:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170176:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098147.649634:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170048:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098147.649670:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417170112:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098147.651181:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098147.651189:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098147.651196:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170176:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098147.837968:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098148.414568:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098148.861872:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098149.436632:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098149.884599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098150.460475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098150.912236:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098151.484795:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098151.932543:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098152.508481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098152.767106:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417170240:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098152.769852:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098152.769859:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098152.769862:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098152.769871:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2656 00000100:00080000:3.0:1766098152.769874:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098152.769880:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098152.769891:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098152.769906:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417170368/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098152.769948:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098152.769952:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098152.769994:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170368:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098152.770101:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417170304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098152.776641:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417170304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098152.776792:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417170240:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098152.776889:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098152.776894:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098152.776899:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170368:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098152.956817:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098153.532485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098153.980752:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098154.556502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098155.004605:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098155.580484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098156.029013:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098156.604754:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098157.052562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098157.628452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098157.886476:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098157.886483:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098157.886487:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098157.886497:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2661 00000100:00080000:1.0:1766098157.886500:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098157.886506:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098157.886516:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098157.886542:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417170560/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098157.886566:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098157.886569:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098157.886600:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417170496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098157.886614:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170560:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098157.886712:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098157.891025:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098157.891035:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098157.891044:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170560:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098157.892449:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417170496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098157.893170:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170432:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098158.076575:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098158.653128:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098159.101484:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098159.676475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098160.124676:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098160.703638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098161.148575:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098161.724647:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098162.172980:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098162.750280:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098163.004592:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170624:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098163.005141:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098163.005179:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417170688:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098163.009267:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170624:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098163.009376:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098163.009382:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098163.009385:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098163.009391:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2666 00000100:00080000:0.0:1766098163.009393:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098163.009399:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098163.009411:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098163.009438:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417170752/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098163.009466:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098163.009469:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098163.009535:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170752:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098163.016701:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417170688:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098163.023565:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098163.023581:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098163.023594:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170752:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098163.200258:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098163.772559:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098164.220658:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098164.796558:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098165.249324:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098165.820527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098166.268722:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098166.846594:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098167.293241:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098167.869801:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098168.124686:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098168.124695:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098168.124699:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098168.124712:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2672 00000100:00080000:1.0:1766098168.124715:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098168.124722:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098168.124735:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098168.124755:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417170944/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098168.124784:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098168.124787:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098168.124823:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417170880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098168.124843:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170944:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098168.124938:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170816:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098168.130899:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098168.130915:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098168.130945:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417170944:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098168.132031:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417170816:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098168.133782:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417170880:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098168.317399:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098168.892516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098169.340924:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098169.916618:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098170.366180:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098170.940442:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098171.388656:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098171.965411:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098172.414401:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098172.988500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098173.244733:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171008:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098173.247769:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098173.247790:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417171072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098173.247864:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098173.247869:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098173.247873:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098173.247878:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2677 00000100:00080000:1.0:1766098173.247881:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098173.247887:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098173.247898:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098173.247920:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417171136/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098173.248007:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171136:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766098173.249306:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098173.249312:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098173.250453:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171008:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:1.0:1766098173.251152:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:0.0:1766098173.251443:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417171072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098173.259884:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098173.259895:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098173.259907:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171136:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098173.438166:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098174.013065:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098174.462093:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098175.036459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098175.484587:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098176.062184:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098176.509180:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098177.084486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098177.533714:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098178.108515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098178.366695:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098178.366704:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098178.366707:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098178.366717:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2682 00000100:00080000:0.0:1766098178.366720:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098178.366726:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098178.366735:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098178.366750:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417171328/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098178.366776:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098178.366780:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098178.366804:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417171264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098178.366809:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171328:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098178.366880:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098178.369867:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098178.369876:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098178.369885:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171328:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098178.372186:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098178.372783:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417171264:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098178.556575:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098179.132582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098179.580762:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098180.156482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098180.606760:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098181.180585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098181.628567:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098182.204471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098182.652556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098183.228475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098183.486841:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098183.486848:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098183.486852:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098183.486861:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2687 00000100:00080000:1.0:1766098183.486864:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098183.486870:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098183.486879:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098183.486892:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f7eb885 x1851885417171520/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098183.486916:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098183.486919:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098183.487746:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171520:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098183.488087:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098183.488124:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098183.488129:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417171456:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098183.495061:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098183.495072:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098183.495084:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417171456:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098183.495084:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171520:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098183.495102:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171392:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098183.676579:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098184.252521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098184.701724:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098185.276466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098185.724562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098186.301325:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098186.748578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098187.324475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098187.772535:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098188.349534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098188.605645:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098188.605652:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098188.605656:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098188.605665:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2692 00000100:00080000:3.0:1766098188.605668:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098188.605674:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098188.605683:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098188.605699:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ebab33a4 x1851885417171712/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098188.605745:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098188.605749:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098188.606034:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417171584:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098188.606339:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098188.606347:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417171648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098188.607324:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171712:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766098188.609970:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098188.609980:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098188.609988:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171712:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098188.611011:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417171648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098188.611024:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417171584:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098188.796516:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098189.372495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098189.820579:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098190.396518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098190.845203:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098191.420511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098191.868595:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098192.444543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098192.893115:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098193.469167:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098193.726345:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098193.730706:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417171840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098193.732077:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417171776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098193.732207:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098193.732212:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098193.732215:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098193.732221:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2697 00000100:00080000:1.0:1766098193.732223:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098193.732228:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098193.732237:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098193.732256:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417171904/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098193.732322:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171904:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766098193.738703:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098193.738715:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098193.738972:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417171840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098193.741615:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098193.741625:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098193.741634:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417171904:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766098193.920178:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098194.492473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098194.941931:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098195.521401:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098195.965437:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098196.540470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098196.989020:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098197.564479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098198.013900:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098198.590205:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098198.845050:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417171968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098198.846004:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098198.846013:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417172032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098198.849272:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098198.849283:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098198.849287:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098198.849297:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2702 00000100:00080000:3.0:1766098198.849300:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098198.849306:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098198.849317:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098198.849347:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417172096/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098198.849390:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098198.849393:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098198.849455:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172096:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098198.858400:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417172032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098198.858419:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417171968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098198.860336:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098198.860346:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098198.860378:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172096:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098199.038301:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098199.612467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098200.060548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098200.636469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098201.085057:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098201.660455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098202.108573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098202.684483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098203.133215:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098203.709082:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098203.964708:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098203.964715:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098203.964719:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098203.964729:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2707 00000100:00100000:1.0:1766098203.964729:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417172160:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098203.964732:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098203.964737:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098203.964747:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098203.964761:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417172288/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098203.964784:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098203.964786:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098203.964806:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417172224:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098203.964881:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172288:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098203.974952:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417172224:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098203.975042:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417172160:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098203.975135:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098203.975140:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098203.975143:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172288:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098204.156825:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098204.734870:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098205.180758:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098205.756952:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098206.207561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098206.780586:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098207.228554:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098207.804467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098208.252569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098208.828475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098209.084797:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098209.084804:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098209.084808:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098209.084818:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2713 00000100:00080000:0.0:1766098209.084827:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098209.084832:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098209.084841:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098209.084857:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52a4842 x1851885417172480/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098209.084873:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098209.084877:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098209.084896:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417172352:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098209.084981:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098209.084987:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417172416:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098209.085019:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172480:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098209.091509:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417172416:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098209.091528:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417172352:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098209.091598:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098209.091602:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098209.091607:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172480:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098209.276649:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098209.855905:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098210.303351:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098210.876491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098211.324580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098211.900599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098212.349073:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098212.924482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098213.378713:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098213.948467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098214.204552:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098214.204560:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098214.204576:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098214.204586:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2718 00000100:00080000:0.0:1766098214.204590:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098214.204596:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098214.204607:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098214.204624:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2c84c91 x1851885417172672/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098214.204653:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098214.204656:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098214.204686:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417172608:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098214.204785:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098214.204792:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417172544:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098214.205005:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172672:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098214.214491:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417172544:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098214.215976:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098214.215988:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098214.215998:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172672:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098214.216125:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417172608:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098214.400096:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098214.972955:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098215.420594:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098215.996549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098216.444772:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098217.020715:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098217.469398:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098218.051510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098218.493391:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098219.070035:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098219.325131:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417172736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098219.325318:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098219.325326:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417172800:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098219.325460:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098219.325466:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098219.325471:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098219.325477:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2723 00000100:00080000:1.0:1766098219.325480:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098219.325485:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098219.325499:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098219.325518:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417172864/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098219.325532:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098219.325535:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098219.325681:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172864:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098219.334302:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417172736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098219.334548:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098219.334555:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098219.334560:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417172864:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098219.334912:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417172800:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098219.519718:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098220.094945:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098220.540982:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098221.117142:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098221.566085:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098222.142917:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098222.588556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098223.164474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098223.612685:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098224.191155:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098224.448634:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098224.448643:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098224.448647:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098224.448658:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2728 00000100:00080000:2.0:1766098224.448662:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098224.448668:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098224.448678:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098224.448693:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417173056/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098224.448716:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098224.448719:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098224.448742:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417172992:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098224.448815:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098224.448820:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417172928:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098224.448899:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173056:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098224.457542:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417172928:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098224.457568:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417172992:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098224.459449:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098224.459458:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098224.459466:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173056:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098224.636525:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098225.212487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098225.660605:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098226.236451:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098226.684744:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098227.260791:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098227.708576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098228.284472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098228.732548:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098229.308480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098229.565627:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098229.565635:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098229.565640:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098229.565648:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2733 00000100:00100000:3.0:1766098229.565648:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417173120:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098229.565652:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098229.565660:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098229.565671:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098229.565703:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417173248/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098229.565727:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098229.565731:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098229.565735:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098229.565744:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417173184:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098229.565961:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173248:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1766098229.572059:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098229.572070:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098229.572079:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173248:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098229.572643:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417173184:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098229.572654:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417173120:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098229.756640:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098230.332530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098230.780559:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098231.358565:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098231.806156:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098232.380880:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098232.828572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098233.404474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098233.853954:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098234.428591:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098234.685139:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417173312:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098234.688661:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098234.688668:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098234.688673:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1766098234.688673:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417173376:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098234.688682:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2738 00000100:00080000:2.0:1766098234.688685:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098234.688690:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098234.688701:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098234.688728:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417173440/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098234.688753:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098234.688756:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098234.688802:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173440:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098234.693000:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417173312:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098234.694797:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098234.694807:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098234.694815:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173440:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098234.695070:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417173376:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098234.878110:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098235.452933:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098235.900891:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098236.476478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098236.925641:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098237.501622:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098237.952429:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098238.525153:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098238.972546:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098239.548486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098239.805768:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098239.805775:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098239.805779:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098239.805789:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2743 00000100:00080000:0.0:1766098239.805793:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098239.805799:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098239.805811:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098239.805827:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000030e2f50e x1851885417173632/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098239.805851:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098239.805855:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098239.805890:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417173568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098239.805999:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098239.806005:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417173504:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098239.806302:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173632:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098239.812491:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417173504:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098239.812512:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417173568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098239.813304:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098239.813311:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098239.813317:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173632:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098239.997169:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098240.575407:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098241.020592:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098241.596494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098242.044578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098242.620476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098243.068555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098243.644480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098244.095347:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098244.668518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098244.924857:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098244.924865:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098244.924868:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098244.924878:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2748 00000100:00080000:0.0:1766098244.924903:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098244.924909:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098244.924920:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098244.924938:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000042205631 x1851885417173824/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098244.924955:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098244.924959:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098244.924989:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173824:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098244.925095:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417173760:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098244.925133:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098244.925141:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417173696:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098244.935162:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417173760:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098244.937320:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417173696:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098244.938892:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098244.938901:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098244.938907:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417173824:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098245.118812:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098245.693423:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098246.144324:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098246.717508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098247.165235:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098247.740471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098248.188604:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098248.764494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098249.214629:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098249.789314:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098250.044595:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098250.044603:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098250.044606:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098250.044616:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2754 00000100:00080000:2.0:1766098250.044618:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098250.044624:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098250.044633:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098250.044648:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417174016/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098250.044674:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098250.044677:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098250.044702:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417173952:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098250.044780:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098250.044786:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417173888:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098250.044892:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174016:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766098250.050520:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098250.050529:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098250.050537:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174016:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098250.052913:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417173888:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098250.055177:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417173952:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098250.237025:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098250.812656:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098251.260532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098251.836437:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098252.284548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098252.860472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098253.309417:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098253.884662:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098254.334045:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098254.908479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098255.165444:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098255.165451:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098255.165454:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098255.165465:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2759 00000100:00080000:1.0:1766098255.165468:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098255.165474:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098255.165485:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098255.165499:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9051765 x1851885417174208/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098255.165543:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098255.165547:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098255.165585:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098255.165631:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417174080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098255.165691:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098255.165698:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417174144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098255.168269:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417174144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098255.168314:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098255.168320:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098255.168325:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098255.173462:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417174080:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098255.359266:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098255.932569:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098256.380623:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098256.956740:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098257.404763:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098257.980521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098258.431064:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098259.004512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098259.453141:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098260.031313:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098260.285329:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417174272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098260.285643:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098260.285650:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417174336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098260.286170:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098260.286178:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098260.286182:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098260.286190:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2764 00000100:00080000:3.0:1766098260.286193:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098260.286200:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098260.286233:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098260.286254:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417174400/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098260.286283:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098260.286287:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098260.286472:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174400:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098260.291731:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417174336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098260.292228:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417174272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098260.295311:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098260.295325:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098260.295337:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174400:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098260.479161:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098261.054981:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098261.501612:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098262.076424:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098262.528147:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098263.100993:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098263.549979:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098264.124533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098264.573093:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098265.148467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098265.404558:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098265.404565:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098265.404568:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098265.404579:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2769 00000100:00080000:3.0:1766098265.404583:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098265.404588:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098265.404599:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098265.404612:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417174592/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098265.404675:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174592:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098265.404862:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417174464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098265.404899:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098265.404904:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417174528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098265.404974:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098265.404978:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1766098265.412743:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098265.412755:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098265.412764:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174592:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098265.420474:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417174464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098265.421556:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417174528:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098265.596571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098266.172481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098266.620770:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098267.196582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098267.644589:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098268.220485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098268.673055:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098269.244494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098269.693430:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098270.268473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098270.524542:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098270.524550:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098270.524553:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098270.524564:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2774 00000100:00080000:3.0:1766098270.524567:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098270.524574:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098270.524583:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098270.524598:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbfb1ac7 x1851885417174784/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098270.524622:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098270.524626:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098270.524666:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098270.524969:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417174720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098270.525021:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098270.525027:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417174656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098270.528706:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098270.528715:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098270.528724:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098270.531318:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417174656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098270.531339:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417174720:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098270.716551:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098271.292459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098271.740782:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098272.317875:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098272.766212:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098273.340488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098273.789257:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098274.364537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098274.814254:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098275.391078:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098275.644563:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098275.644571:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098275.644574:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098275.644585:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2779 00000100:00080000:3.0:1766098275.644588:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098275.644594:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098275.644605:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098275.644634:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004951ed42 x1851885417174976/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098275.644668:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098275.644672:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098275.644776:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417174848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098275.644872:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098275.644879:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417174912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098275.645784:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174976:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098275.654584:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417174848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098275.654631:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417174912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098275.654828:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098275.654833:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098275.654838:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417174976:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098275.839638:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098276.412927:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098276.861467:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098277.436503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098277.890298:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098278.464256:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098278.908581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098279.486759:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098279.932589:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098280.508446:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098280.766963:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098280.766970:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098280.766974:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098280.766985:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2784 00000100:00080000:1.0:1766098280.766989:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098280.767007:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098280.767018:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098280.767034:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417175168/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098280.767048:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098280.767050:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098280.767125:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175040:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098280.767218:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098280.767225:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175104:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098280.767259:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175168:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1766098280.773568:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098280.773579:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098280.773588:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175168:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098280.775263:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175104:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098280.775280:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175040:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098280.956555:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098281.532469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098281.980937:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098282.556505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098283.004559:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098283.580566:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098284.035005:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098284.605580:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098285.053093:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098285.630864:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098285.884621:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098285.884628:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098285.884632:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098285.884641:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2789 00000100:00100000:1.0:1766098285.884642:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175232:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098285.884645:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098285.884653:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098285.884663:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098285.884689:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417175360/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098285.884713:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098285.884728:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098285.884746:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417175296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098285.884785:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175360:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098285.891376:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417175296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098285.894391:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098285.894402:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098285.894410:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175360:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098285.894663:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175232:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098286.077178:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098286.655397:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098287.110247:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098287.676464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098288.125785:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098288.700480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098289.150044:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098289.724443:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098290.172578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098290.748567:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098291.004710:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098291.004731:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098291.004735:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098291.004745:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2794 00000100:00080000:1.0:1766098291.004748:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098291.004754:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098291.004763:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098291.004777:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417175552/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098291.004799:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098291.004803:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098291.004862:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175552:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098291.005214:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175424:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098291.005249:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098291.005254:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175488:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098291.012660:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098291.012671:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098291.012680:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175552:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098291.013041:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175488:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098291.013052:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175424:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098291.196476:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098291.772493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098292.220587:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098292.797103:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098293.247704:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098293.820484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098294.268629:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098294.844431:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098295.293750:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098295.868473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098296.124540:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098296.124548:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098296.124552:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098296.124565:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2800 00000100:00080000:1.0:1766098296.124568:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098296.124575:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098296.124585:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098296.124607:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417175744/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098296.124653:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098296.124657:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098296.124749:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175744:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098296.125081:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175616:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098296.125132:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098296.125140:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098296.135015:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098296.135029:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098296.135038:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175744:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098296.135772:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098296.135787:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175616:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098296.317124:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098296.892592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098297.343724:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098297.916499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098298.364578:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098298.940573:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098299.392606:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098299.964479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098300.415624:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098300.989696:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098301.244559:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098301.244565:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098301.244568:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098301.244578:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2805 00000100:00080000:1.0:1766098301.244580:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098301.244890:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098301.244900:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098301.244915:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417175936/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098301.244981:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175936:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098301.245490:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175808:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098301.245531:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098301.245537:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175872:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098301.246088:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098301.246094:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098301.249278:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417175872:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098301.249294:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417175808:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098301.250071:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098301.250077:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098301.250084:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417175936:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098301.436722:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098302.012517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098302.462345:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098303.036485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098303.486530:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098304.061965:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098304.508632:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098305.084503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098305.532551:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098306.108507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098306.364613:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098306.364621:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098306.364624:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098306.364634:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2810 00000100:00080000:3.0:1766098306.364637:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098306.364642:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098306.364652:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098306.364667:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417176128/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098306.364716:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098306.364719:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098306.364766:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176128:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098306.364811:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417176000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098306.364994:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098306.365001:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417176064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098306.374060:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417176000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098306.374381:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098306.374386:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098306.374391:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176128:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098306.374744:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417176064:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098306.556594:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098307.132488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098307.580607:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098308.156509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098308.604542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098309.180484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098309.630762:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098310.208101:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098310.653914:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098311.228605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098311.484625:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098311.484633:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098311.484636:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098311.484648:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2815 00000100:00080000:1.0:1766098311.484651:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098311.484656:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098311.484666:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098311.484691:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417176320/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098311.484715:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098311.484718:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098311.484802:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417176256:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098311.484896:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098311.484901:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417176192:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098311.484982:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176320:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098311.487569:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417176192:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098311.487590:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417176256:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098311.488282:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098311.488288:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098311.488293:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176320:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098311.679152:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098312.253610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098312.702812:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098313.276489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098313.725297:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098314.306225:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098314.748606:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098315.324479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098315.772561:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098316.348474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098316.604523:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098316.604530:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098316.604554:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098316.604564:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2820 00000100:00100000:0.0:1766098316.604566:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417176384:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098316.604568:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098316.604581:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098316.604592:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098316.604609:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417176512/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098316.604627:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098316.604630:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098316.604661:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098316.604670:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417176448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098316.614678:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176512:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098316.617989:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417176384:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098316.618649:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098316.618661:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098316.618670:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176512:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098316.619778:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417176448:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098316.796555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098317.372495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098317.820690:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098318.396415:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098318.844585:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098319.420466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098319.868553:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098320.444477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098320.892539:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098321.468590:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098321.727015:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417176576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098321.728071:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098321.728082:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417176640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098321.728186:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098321.728191:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098321.728194:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098321.728199:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2825 00000100:00080000:0.0:1766098321.728202:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098321.728222:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098321.728231:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098321.728249:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417176704/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098321.728271:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098321.728275:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098321.728504:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176704:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766098321.734683:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098321.734694:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098321.734704:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176704:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098321.739128:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417176640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098321.739144:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417176576:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098321.918102:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098322.492628:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098322.940555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098323.516491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098323.964544:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098324.541750:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098324.988793:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098325.564547:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098326.012905:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098326.589595:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098326.844937:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098326.844945:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098326.844948:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098326.844958:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2830 00000100:00080000:1.0:1766098326.844961:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098326.844967:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098326.844976:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098326.844994:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417176896/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098326.845018:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098326.845022:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098326.845042:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417176832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098326.845128:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098326.845134:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417176768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098326.846056:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098326.849686:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417176768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098326.849705:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417176832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1766098326.851212:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:0.0:1766098326.859585:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098326.859597:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098326.859608:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417176896:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098327.036589:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098327.614399:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098328.060813:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098328.637282:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098329.085450:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098329.661975:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098330.108594:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098330.685691:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098331.132567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098331.710424:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098331.964531:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098331.964537:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098331.964540:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098331.964549:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2835 00000100:00080000:3.0:1766098331.964552:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098331.964557:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098331.964565:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098331.964577:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417177088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098331.964597:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098331.964600:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098331.964620:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417177024:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098331.964700:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098331.964705:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417176960:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098331.964785:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177088:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098331.969433:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417177024:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098331.969461:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417176960:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098331.972554:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098331.972565:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098331.972574:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177088:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098332.157142:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098332.733163:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098333.180740:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098333.757900:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098334.206189:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098334.780482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098335.228645:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098335.804522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098336.252514:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098336.828488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098337.084540:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098337.084546:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098337.084550:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098337.084559:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2841 00000100:00080000:3.0:1766098337.084562:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098337.084568:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098337.084578:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098337.084591:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417177280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098337.084612:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098337.084615:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098337.084706:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417177152:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098337.084805:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098337.084811:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417177216:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098337.085136:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177280:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098337.088947:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417177152:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098337.089272:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098337.089277:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098337.089282:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177280:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098337.093951:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417177216:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098337.279022:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098337.852422:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098338.302330:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098338.876470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098339.326855:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098339.900484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098340.349122:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098340.925234:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098341.375677:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098341.948471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098342.204561:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417177344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098342.207782:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417177344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098342.208293:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417177408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098342.214768:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098342.214779:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098342.214783:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098342.214793:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2846 00000100:00080000:1.0:1766098342.214797:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098342.214815:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098342.214826:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098342.214853:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417177472/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098342.214869:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098342.214873:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098342.215000:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177472:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098342.219722:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417177408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098342.219824:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098342.219828:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098342.219832:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177472:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098342.396552:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098342.972462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098343.420583:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098343.996479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098344.446281:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098345.020470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098345.468658:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098346.044430:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098346.494372:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098347.069040:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098347.328344:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098347.328397:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098347.328401:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098347.328412:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2851 00000100:00080000:3.0:1766098347.328415:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098347.328421:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098347.328431:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098347.328448:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417177664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098347.328475:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098347.328479:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098347.328502:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417177600:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098347.328577:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098347.328584:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417177536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098347.329175:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177664:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098347.331866:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417177536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098347.332796:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417177600:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098347.334276:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098347.334284:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098347.334291:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177664:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098347.517383:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098348.092471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098348.540536:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098349.116767:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098349.567480:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098350.140470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098350.588693:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098351.164485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098351.612741:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098352.188474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098352.445036:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098352.445043:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098352.445047:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098352.445059:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2856 00000100:00080000:0.0:1766098352.445063:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098352.445068:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098352.445079:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098352.445109:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005ed5ddd4 x1851885417177856/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098352.445130:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098352.445134:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098352.445282:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417177792:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098352.445388:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098352.445393:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417177728:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098352.445922:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177856:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766098352.458194:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098352.458206:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098352.458215:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417177856:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098352.458258:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417177728:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098352.458276:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417177792:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098352.636593:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098353.212473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098353.660529:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098354.236479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098354.685039:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098355.260467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098355.708567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098356.284468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098356.733037:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098357.308476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098357.564634:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098357.564642:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1766098357.564651:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417177920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098357.564660:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098357.564667:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2861 00000100:00080000:2.0:1766098357.564670:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098357.564676:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098357.564686:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098357.564704:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417178048/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098357.564729:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098357.564733:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098357.564739:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098357.564748:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417177984:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098357.568141:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178048:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098357.568795:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417177984:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098357.571611:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417177920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098357.572579:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098357.572586:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098357.572593:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178048:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098357.757178:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098358.334881:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098358.780525:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766098359.356471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098359.805865:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098360.391462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098360.828556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098361.404485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098361.852545:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098362.428478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098362.684489:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098362.684498:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098362.684501:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098362.684511:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2866 00000100:00080000:2.0:1766098362.684515:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098362.684520:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098362.684530:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098362.684546:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417178240/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098362.684568:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098362.684571:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098362.684594:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417178176:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098362.684690:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098362.684697:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417178112:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098362.685220:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178240:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766098362.690947:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098362.690960:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098362.690970:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178240:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098362.691490:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417178176:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098362.694198:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417178112:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098362.876549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098363.452493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098363.901606:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098364.479307:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098364.926089:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098365.502053:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098365.953583:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098366.524469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098366.973401:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098367.548458:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098367.804728:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098367.804735:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098367.804738:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098367.804748:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2871 00000100:00080000:0.0:1766098367.804751:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098367.804757:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098367.804766:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098367.804782:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000512a78fc x1851885417178432/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098367.804803:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098367.804806:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098367.804845:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178432:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098367.804874:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417178304:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098367.804933:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098367.804939:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417178368:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098367.812731:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417178368:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098367.813673:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098367.813679:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098367.813698:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178432:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098367.813881:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417178304:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098367.998194:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098368.576005:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098369.020539:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098369.596549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098370.047205:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098370.620465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098371.068665:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098371.644474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098372.092555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098372.668471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098372.925053:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098372.925060:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098372.925064:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098372.925073:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2876 00000100:00080000:0.0:1766098372.925076:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098372.925081:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098372.925089:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098372.925104:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000024975221 x1851885417178624/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098372.925132:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098372.925135:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098372.925213:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417178560:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098372.926029:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178624:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098372.926144:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098372.926150:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417178496:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098372.929517:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417178560:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098372.931072:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098372.931082:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098372.931090:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178624:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098372.932331:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417178496:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098373.116738:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098373.692431:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098374.143778:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098374.716472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098375.166661:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098375.740471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098376.188938:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098376.764571:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098377.214459:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098377.788638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098378.045141:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098378.045149:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098378.045152:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098378.045162:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2882 00000100:00080000:1.0:1766098378.045165:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098378.045171:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098378.045182:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098378.045213:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000653e2b14 x1851885417178816/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098378.045237:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098378.045241:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098378.045277:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417178752:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098378.045391:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098378.045405:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417178688:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098378.045731:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178816:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098378.048733:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417178688:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098378.048854:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098378.048859:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098378.048882:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417178816:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098378.051157:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417178752:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098378.241701:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098378.812545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098379.265636:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098379.836553:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098380.285954:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098380.860667:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098381.310969:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098381.885955:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098382.337556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098382.910979:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098383.164786:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417178880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098383.166432:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098383.166440:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098383.166444:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098383.166455:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2887 00000100:00080000:3.0:1766098383.166477:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098383.166483:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098383.166497:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098383.166516:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417179008/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098383.166540:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098383.166544:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098383.166568:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417178944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098383.166802:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179008:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098383.170339:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417178880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098383.180745:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098383.180757:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098383.180766:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179008:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098383.181111:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417178944:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098383.366295:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098383.934946:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098384.380970:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098384.957639:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098385.404569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098385.983410:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098386.428740:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098387.004425:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098387.452569:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098388.028560:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098388.285125:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098388.285132:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098388.285136:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098388.285146:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2892 00000100:00080000:0.0:1766098388.285149:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098388.285154:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098388.285171:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098388.285187:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417179200/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098388.285214:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098388.285217:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098388.285239:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417179072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098388.285318:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098388.285324:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417179136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098388.285560:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179200:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098388.289301:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417179072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098388.291068:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417179136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098388.297798:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098388.297809:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098388.297826:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179200:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098388.476608:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098389.052583:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098389.500616:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098390.076489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098390.524566:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098391.100462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098391.548613:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098392.124421:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098392.572647:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098393.148468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098393.404701:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098393.404709:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098393.404712:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098393.404733:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2897 00000100:00080000:1.0:1766098393.404736:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098393.404742:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098393.404752:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098393.404766:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417179392/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098393.404827:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179392:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1766098393.404924:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098393.404928:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098393.404943:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417179328:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098393.404978:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098393.404982:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417179264:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098393.413273:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098393.413283:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098393.413305:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179392:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098393.413947:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417179264:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098393.413963:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417179328:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098393.597403:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098394.172704:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098394.620600:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098395.196480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098395.644542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098396.220822:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098396.671744:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098397.244862:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098397.693027:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098398.268573:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098398.524591:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098398.524598:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098398.524601:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098398.524611:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2902 00000100:00080000:0.0:1766098398.524613:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098398.524619:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098398.524627:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098398.524641:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000030e2f50e x1851885417179584/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098398.524660:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098398.524663:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098398.524712:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098398.524753:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417179456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098398.524821:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098398.524830:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417179520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098398.534140:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417179456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098398.536449:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098398.536461:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098398.536473:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098398.536925:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417179520:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098398.717399:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098399.292487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098399.740546:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098400.316846:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098400.764549:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098401.340612:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098401.788557:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098402.365425:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098402.814238:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098403.388482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098403.644623:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098403.644630:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098403.644633:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098403.644643:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2907 00000100:00080000:1.0:1766098403.644646:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098403.644650:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098403.644659:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098403.644675:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417179776/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098403.644709:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098403.644713:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098403.644734:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417179712:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098403.644823:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098403.644830:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417179648:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098403.649584:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179776:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098403.649813:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417179648:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098403.649881:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417179712:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098403.652895:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098403.652905:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098403.652914:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179776:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098403.836868:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098404.412483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098404.860610:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098405.436512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098405.885982:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098406.461128:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098406.910746:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098407.484792:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098407.932571:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098408.508862:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098408.765580:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098408.765588:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098408.765591:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098408.765602:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2912 00000100:00080000:2.0:1766098408.765605:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098408.765610:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098408.765620:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098408.765634:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ba671fa0 x1851885417179968/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098408.765658:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098408.765661:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098408.765697:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417179904:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098408.765786:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098408.765793:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417179840:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098408.765917:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179968:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098408.775419:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417179904:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098408.775465:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417179840:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098408.775677:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098408.775682:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098408.775686:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417179968:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098408.957066:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098409.532462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098409.980682:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098410.556782:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098411.005970:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098411.580492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098412.033394:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098412.604528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098413.052711:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098413.628455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098413.886801:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098413.886808:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098413.886810:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098413.886820:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2917 00000100:00080000:1.0:1766098413.886823:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098413.886828:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098413.886837:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098413.886849:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000653e2b14 x1851885417180160/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098413.886875:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098413.886878:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098413.886900:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417180096:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098413.886990:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098413.886996:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417180032:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098413.887640:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180160:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098413.892283:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417180096:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098413.894234:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417180032:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098413.896758:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098413.896769:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098413.896776:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180160:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098414.077146:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098414.654863:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098415.100561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098415.678832:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098416.124547:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098416.700492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098417.149133:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098417.725313:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098418.174918:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098418.748488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098419.004671:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098419.004680:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098419.004684:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098419.004696:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2922 00000100:00080000:0.0:1766098419.004699:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098419.004706:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098419.004717:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098419.004734:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417180352/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098419.004753:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098419.004756:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098419.004831:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417180224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098419.004953:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098419.004959:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417180288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098419.005629:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098419.017576:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417180224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098419.019671:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098419.019682:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098419.019689:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098419.019964:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417180288:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098419.196540:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098419.772463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098420.222932:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098420.796477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098421.245340:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098421.821030:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098422.270141:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098422.844476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098423.293552:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098423.871083:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098424.124673:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098424.124684:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098424.124701:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766098424.124710:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417180416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098424.124716:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2928 00000100:00080000:3.0:1766098424.124721:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098424.124732:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098424.124765:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098424.124793:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417180544/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098424.124822:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098424.124832:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417180480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098424.124884:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180544:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766098424.126965:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098424.126974:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098424.127099:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417180480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098424.127681:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417180416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098424.128031:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098424.128037:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098424.128043:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180544:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098424.317269:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098424.893433:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098425.342557:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098425.916503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098426.364659:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098426.940554:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098427.388627:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098427.964487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098428.412584:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098428.989219:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098429.244776:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1766098429.244777:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417180608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098429.244796:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098429.244800:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098429.244808:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2933 00000100:00080000:0.0:1766098429.244831:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098429.244839:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098429.244854:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098429.244877:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417180736/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098429.244901:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098429.244909:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417180672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098429.244938:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766098429.245151:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098429.245155:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766098429.262907:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098429.262923:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098429.262936:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098429.263257:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417180672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098429.263271:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417180608:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098429.436629:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098430.012531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098430.464581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098431.036545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098431.488911:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098432.060955:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098432.509809:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098433.084613:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098433.532627:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098434.109445:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098434.364713:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098434.364724:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098434.364728:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098434.364743:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2938 00000100:00080000:1.0:1766098434.364746:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098434.364753:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098434.364767:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098434.364785:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417180928/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098434.364811:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098434.364815:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098434.364849:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417180864:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098434.364970:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098434.364976:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417180800:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098434.365120:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180928:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766098434.369495:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098434.369511:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098434.369522:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417180928:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098434.369659:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417180800:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098434.369679:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417180864:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098434.557540:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098435.132890:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098435.586003:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098436.160593:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098436.607227:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098437.181340:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098437.631728:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098438.204557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098438.658204:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098439.229604:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098439.484637:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098439.484644:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098439.484647:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098439.484657:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2943 00000100:00100000:2.0:1766098439.484671:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417180992:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098439.484674:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098439.484683:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098439.484693:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098439.484712:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417181120/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098439.484736:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098439.484740:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098439.484760:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098439.484768:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417181056:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098439.484975:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181120:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098439.492675:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417180992:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098439.493957:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098439.493965:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098439.493971:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181120:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098439.494043:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417181056:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098439.676580:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098440.253070:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098440.700762:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098441.276685:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098441.724575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098442.300508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098442.748575:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098443.324492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098443.772562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098444.348486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098444.604605:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098444.604628:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098444.604632:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766098444.604643:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417181184:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098444.604643:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2948 00000100:00080000:1.0:1766098444.604646:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098444.604709:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098444.604720:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098444.604741:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002600e542 x1851885417181312/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098444.604764:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098444.604767:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098444.604787:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417181248:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098444.604826:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181312:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098444.614576:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417181248:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098444.615800:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417181184:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098444.615962:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098444.615967:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098444.615972:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181312:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098444.797421:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098445.374037:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098445.821414:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098446.396538:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098446.845756:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098447.420537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098447.868600:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098448.444491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098448.892675:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098449.468650:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098449.727564:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098449.727571:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098449.727574:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098449.727583:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2953 00000100:00080000:3.0:1766098449.727586:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098449.727605:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098449.727614:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098449.727638:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417181504/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098449.727659:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098449.727662:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098449.727685:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417181440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098449.727714:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181504:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098449.727716:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417181376:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098449.742146:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417181376:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098449.743983:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098449.743993:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098449.744017:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181504:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098449.744116:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417181440:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098449.916906:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098450.494772:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098450.940584:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098451.517330:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098451.965585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098452.541614:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098452.989213:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098453.564890:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098454.012913:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098454.588823:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098454.845409:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098454.845417:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098454.845420:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098454.845431:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2958 00000100:00080000:0.0:1766098454.845434:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098454.845440:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098454.845450:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098454.845465:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d602f5b x1851885417181696/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098454.845539:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181696:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098454.847512:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417181568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098454.847686:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098454.847691:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098454.847738:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417181632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098454.851792:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417181632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098454.851902:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098454.851907:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098454.851911:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181696:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098454.853787:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417181568:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098455.036797:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098455.612421:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098456.063405:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098456.636478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098457.084575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098457.660858:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098458.108586:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098458.684482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098459.132566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098459.708957:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098459.964808:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098459.964815:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098459.964819:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098459.964829:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2963 00000100:00080000:3.0:1766098459.964832:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098459.964837:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098459.964860:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098459.964879:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417181888/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098459.964908:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098459.964912:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098459.964977:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181888:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098459.966647:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417181760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098459.966727:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098459.966734:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417181824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098459.972953:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098459.972964:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098459.972973:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417181888:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098459.980254:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417181824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098459.982145:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417181760:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098460.159535:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098460.732952:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098461.180507:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098461.758186:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098462.214585:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098462.780522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098463.228612:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098463.804493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098464.255291:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098464.828518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098465.084515:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098465.084522:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098465.084526:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098465.084538:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2969 00000100:00080000:1.0:1766098465.084541:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098465.084547:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098465.084557:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098465.084560:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417181952:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098465.084572:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000baf3fdb7 x1851885417182080/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098465.084595:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098465.084599:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098465.084630:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417182016:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098465.084649:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182080:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098465.097880:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417181952:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098465.100046:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417182016:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098465.100229:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098465.100234:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098465.100239:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182080:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098465.276590:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098465.852482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098466.300533:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098466.876555:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098467.326610:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098467.900477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098468.349501:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098468.925386:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098469.372716:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098469.948478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098470.205034:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098470.205042:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098470.205045:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098470.205056:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2974 00000100:00080000:3.0:1766098470.205059:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098470.205065:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098470.205075:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098470.205090:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417182272/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098470.205132:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098470.205135:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098470.205226:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182272:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098470.206314:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417182208:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098470.207349:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098470.207401:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417182144:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098470.211585:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098470.211598:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098470.211609:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182272:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098470.217506:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417182144:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098470.217528:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417182208:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098470.397137:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098470.972607:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098471.422275:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098471.999224:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098472.444537:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098473.020470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098473.468547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098474.044460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098474.494516:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098475.068486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098475.324624:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417182336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098475.327084:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098475.327101:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417182400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098475.329764:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098475.329789:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098475.329793:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098475.329803:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2979 00000100:00080000:1.0:1766098475.329807:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098475.329813:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098475.329825:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098475.329849:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417182464/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098475.329917:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182464:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098475.330977:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417182336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098475.331007:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417182400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098475.331642:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098475.331646:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1766098475.335623:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098475.335634:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098475.335643:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182464:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098475.517630:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098476.092933:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098476.540683:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098477.117399:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098477.564614:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098478.140704:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098478.589456:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098479.164442:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098479.612571:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098480.189993:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098480.444551:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098480.444580:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098480.444583:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098480.444593:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2984 00000100:00080000:1.0:1766098480.444596:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098480.444601:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098480.444611:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098480.444624:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417182656/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098480.444645:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098480.444660:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098480.444909:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182656:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098480.445100:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417182592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098480.445153:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098480.445160:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417182528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098480.454780:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417182592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098480.454820:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417182528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:2.0:1766098480.455300:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:2.0:1766098480.455313:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098480.455318:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098480.455322:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182656:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098480.638583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098481.212529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098481.661490:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098482.236529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098482.684659:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098483.261267:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098483.708612:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098484.285063:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098484.733534:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098485.308554:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098485.564615:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098485.564622:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098485.564626:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098485.564636:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2989 00000100:00080000:2.0:1766098485.564640:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098485.564646:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098485.564655:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098485.564671:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b83cf9b1 x1851885417182848/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098485.564712:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098485.564716:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098485.564753:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417182784:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098485.564842:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098485.564848:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417182720:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098485.567227:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182848:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098485.572079:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417182784:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098485.572120:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417182720:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098485.576017:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098485.576033:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098485.576046:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417182848:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098485.756553:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098486.332663:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098486.781380:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098487.356507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098487.805209:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098488.380467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098488.829016:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098489.404464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098489.853443:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098490.432259:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098490.688319:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098490.688327:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098490.688330:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098490.688343:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2994 00000100:00080000:1.0:1766098490.688346:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098490.688351:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098490.688414:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098490.688433:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417183040/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098490.688458:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098490.688461:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098490.688548:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417182976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098490.688644:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098490.688651:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417182912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098490.688660:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183040:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098490.691639:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417182912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098490.691642:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098490.691648:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098490.691655:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183040:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098490.691667:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417182976:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098490.876541:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098491.452468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098491.901381:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098492.476476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098492.927895:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098493.500837:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098493.948579:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098494.524506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098494.972538:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098495.550245:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098495.808431:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098495.808439:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098495.808443:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098495.808454:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 2999 00000100:00080000:0.0:1766098495.808457:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1766098495.808468:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417183104:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098495.808480:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098495.808490:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098495.808509:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004229f3d5 x1851885417183232/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098495.808523:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098495.808526:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098495.808543:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417183168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098495.809014:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183232:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098495.814605:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417183104:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098495.815823:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417183168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098495.815953:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098495.815958:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098495.815962:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183232:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098495.997526:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098496.572478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098497.031480:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098497.596470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098498.044475:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098498.620483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098499.068541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098499.644487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098500.092532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098500.668523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098500.924723:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098500.924730:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098500.924734:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098500.924744:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3004 00000100:00080000:3.0:1766098500.924747:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098500.924753:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098500.924763:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098500.924776:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417183424/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098500.924802:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098500.924804:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098500.924827:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417183360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098500.924899:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098500.924904:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417183296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098500.924954:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183424:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098500.926992:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417183296:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098500.927473:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417183360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098500.943939:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098500.943948:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098500.943956:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183424:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098501.117518:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098501.698138:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098502.140549:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098502.716762:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098503.164582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098503.741917:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098504.189406:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098504.764527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098505.212514:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098505.788454:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098506.049262:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098506.049270:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098506.049281:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098506.049292:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3010 00000100:00080000:3.0:1766098506.049295:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098506.049302:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766098506.049308:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417183488:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:3.0:1766098506.049327:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098506.049346:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417183616/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098506.049393:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098506.049396:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098506.049420:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098506.049427:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417183552:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098506.050116:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183616:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098506.053290:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417183552:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098506.053648:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098506.053653:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098506.053657:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183616:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766098506.053879:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417183488:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098506.237379:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098506.812510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098507.260557:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098507.836514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098508.284557:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098508.860458:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098509.310798:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098509.884489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098510.333118:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098510.908436:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098511.164823:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098511.164830:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098511.164834:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098511.164844:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3015 00000100:00080000:1.0:1766098511.164847:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098511.164852:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098511.164861:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098511.164875:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417183808/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098511.164899:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098511.164911:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098511.164992:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417183680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098511.165083:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098511.165091:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417183744:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098511.165118:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183808:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098511.166953:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417183744:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098511.167504:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417183680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098511.170412:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098511.170435:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098511.170443:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417183808:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098511.358400:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098511.941807:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098512.380559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098512.956437:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098513.404631:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098513.980477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098514.429171:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098515.004950:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098515.454279:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098516.028458:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098516.284551:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098516.284560:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098516.284563:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098516.284573:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3020 00000100:00080000:1.0:1766098516.284577:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098516.284583:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098516.284594:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098516.284611:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417184000/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098516.284641:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098516.284645:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098516.284671:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417183936:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098516.284771:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098516.284778:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417183872:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098516.285155:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184000:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098516.293423:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417183936:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098516.294564:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417183872:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098516.298843:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098516.298856:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098516.298896:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184000:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098516.480708:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098517.052737:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098517.505145:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098518.076539:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098518.525092:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098519.100754:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098519.548835:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098520.124472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098520.574216:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098521.148645:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098521.404622:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098521.404630:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098521.404634:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098521.404646:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3025 00000100:00100000:0.0:1766098521.404648:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417184064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098521.404650:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098521.404657:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098521.404669:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098521.404685:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417184192/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098521.404709:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098521.404712:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098521.404733:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417184128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098521.406317:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184192:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098521.411641:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417184064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098521.413921:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417184128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098521.416024:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098521.416036:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098521.416046:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184192:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766098521.596686:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098522.172543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098522.621405:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098523.196549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098523.645280:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098524.220513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098524.673078:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098525.245021:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098525.693115:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098526.268606:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098526.530942:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098526.530949:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098526.530952:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098526.530962:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3030 00000100:00080000:2.0:1766098526.530965:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098526.530969:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098526.530978:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098526.530994:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009119b13e x1851885417184384/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098526.531014:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098526.531018:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098526.531045:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417184320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098526.531122:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098526.531127:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417184256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098526.531246:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184384:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766098526.536134:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098526.536149:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098526.536161:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184384:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098526.536255:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417184256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098526.536272:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417184320:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098526.716785:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098527.292485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098527.741875:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098528.318256:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098528.764944:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098529.340632:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098529.788642:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098530.364520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098530.813816:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098531.388632:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098531.646892:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098531.646901:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098531.646906:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098531.646920:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3035 00000100:00080000:1.0:1766098531.646924:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098531.646931:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098531.646945:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098531.646964:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417184576/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098531.646998:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098531.647002:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098531.647034:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417184448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098531.647146:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098531.647153:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417184512:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098531.647923:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184576:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766098531.656698:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098531.656706:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098531.656714:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184576:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098531.660446:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417184512:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098531.660493:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417184448:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098531.836732:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098532.412480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098532.861304:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098533.437020:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098533.884551:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098534.460655:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098534.912313:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098535.484494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098535.932736:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098536.509318:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098536.765247:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417184640:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098536.768045:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098536.768052:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098536.768055:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098536.768063:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3040 00000100:00080000:2.0:1766098536.768079:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098536.768084:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098536.768095:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098536.768110:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417184768/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098536.768134:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098536.768137:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098536.768157:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098536.768169:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417184704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098536.768259:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184768:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098536.770601:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417184640:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098536.776838:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417184704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098536.777233:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098536.777244:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098536.777252:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184768:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766098536.956612:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098537.533385:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098537.981413:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098538.556471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098539.004517:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098539.580476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098540.028572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098540.604476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098541.052501:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098541.628528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098541.885142:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098541.885151:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098541.885154:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098541.885164:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3045 00000100:00080000:1.0:1766098541.885168:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098541.885174:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098541.885184:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098541.885197:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9051765 x1851885417184960/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098541.885224:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098541.885227:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098541.885254:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417184896:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098541.885386:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184960:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098541.887377:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098541.887396:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417184832:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098541.892550:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417184832:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098541.892589:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417184896:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098541.893023:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098541.893028:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098541.893032:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417184960:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098542.077747:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098542.653400:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098543.100597:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098543.676520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098544.124575:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098544.700495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098545.149624:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098545.725410:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098546.172572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098546.750687:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098547.004571:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098547.004577:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098547.004581:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098547.004591:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3050 00000100:00080000:2.0:1766098547.004594:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098547.004600:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098547.004610:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098547.004624:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007b4dad67 x1851885417185152/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098547.004668:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098547.004671:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098547.004707:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417185088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098547.004791:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098547.004796:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417185024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098547.005293:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185152:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098547.011581:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417185024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098547.011604:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417185088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098547.015865:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098547.015877:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098547.015902:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185152:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766098547.196558:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098547.773146:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098548.220749:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098548.796488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098549.244650:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098549.820453:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098550.269211:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098550.844659:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098551.297020:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098551.868981:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098552.124675:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098552.124684:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098552.124688:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098552.124702:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3056 00000100:00080000:0.0:1766098552.124706:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098552.124714:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098552.124727:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098552.124747:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023592e07 x1851885417185344/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098552.124771:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098552.124774:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098552.124876:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417185216:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098552.124944:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185344:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098552.126054:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417185280:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098552.135094:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417185280:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098552.135590:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098552.135596:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098552.135611:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185344:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098552.135724:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417185216:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098552.316729:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098552.892522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098553.340600:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098553.916494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098554.364599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098554.940469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098555.388572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098555.964639:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098556.412601:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098556.988791:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098557.244584:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417185408:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098557.269475:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098557.269482:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098557.269496:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098557.269507:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3061 00000100:00080000:3.0:1766098557.269510:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098557.269516:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098557.269525:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098557.269541:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417185536/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098557.269556:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098557.269559:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098557.269591:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417185472:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098557.269920:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185536:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098557.270193:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417185408:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098557.272273:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417185472:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098557.273886:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098557.273893:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098557.273900:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185536:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098557.436583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098558.012788:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098558.461116:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098559.036714:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098559.484586:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098560.060574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098560.508549:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098561.085496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098561.534499:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098562.108629:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098562.367314:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098562.367321:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098562.367324:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098562.367336:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3066 00000100:00080000:0.0:1766098562.367339:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098562.367344:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098562.367373:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098562.367388:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009cc3c5f5 x1851885417185728/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098562.367410:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098562.367413:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098562.367436:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417185664:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098562.367436:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417185600:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098562.367635:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185728:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098562.373073:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417185664:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098562.374912:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098562.374923:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098562.374931:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185728:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098562.377575:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417185600:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098562.557096:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098563.132659:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098563.580611:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098564.156477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098564.606712:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098565.180466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098565.628655:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098566.204464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098566.652582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098567.229625:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098567.484785:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098567.484793:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098567.484797:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098567.484809:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3071 00000100:00080000:1.0:1766098567.484812:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098567.484818:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098567.484829:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098567.484845:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417185920/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098567.484873:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098567.484877:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098567.484901:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185920:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098567.485129:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417185792:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098567.485170:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098567.485176:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417185856:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098567.496776:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417185856:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098567.496796:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417185792:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098567.496904:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098567.496909:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098567.496913:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417185920:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098567.677675:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098568.252473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098568.700792:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098569.276461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098569.724582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098570.300681:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098570.749338:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098571.324514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098571.774949:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098572.348487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098572.606705:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098572.606712:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098572.606715:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098572.606724:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3076 00000100:00080000:3.0:1766098572.606727:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098572.606733:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098572.606743:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098572.606759:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417186112/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098572.606784:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098572.606788:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098572.606824:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417186048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098572.606901:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098572.606921:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417185984:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098572.607727:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186112:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766098572.614533:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1766098572.614544:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417186048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098572.614546:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098572.614559:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186112:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098572.616531:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417185984:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098572.796636:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098573.373256:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098573.822082:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098574.397833:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098574.844644:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098575.424184:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098575.869031:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098576.448599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098576.893468:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098577.472523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098577.724730:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417186176:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098577.725519:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098577.725529:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417186240:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098577.725949:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098577.725956:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098577.725960:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098577.725967:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3081 00000100:00080000:1.0:1766098577.725970:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098577.725977:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098577.725991:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098577.726012:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417186304/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098577.726083:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186304:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098577.728220:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417186176:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098577.728311:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098577.728317:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098577.741616:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417186240:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098577.744820:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098577.744834:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098577.744848:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186304:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766098577.917973:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098578.492536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098578.940989:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098579.516549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098579.964844:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098580.540532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098580.988657:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098581.564553:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098582.014727:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098582.588566:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098582.844787:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098582.844794:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098582.844798:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098582.844808:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3086 00000100:00080000:0.0:1766098582.844811:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098582.844817:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098582.844827:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098582.844844:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417186496/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098582.844866:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098582.844870:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098582.844918:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417186368:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098582.844962:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186496:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098582.845014:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098582.845028:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417186432:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098582.856616:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098582.856631:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098582.856644:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186496:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098582.860510:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417186432:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098582.861984:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417186368:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098583.037253:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098583.612530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098584.060637:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098584.637099:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098585.084601:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098585.660798:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098586.109174:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098586.684527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098587.135632:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098587.708922:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098587.964589:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098587.964598:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098587.964602:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098587.964616:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3091 00000100:00080000:0.0:1766098587.964620:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098587.964627:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098587.964640:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098587.964659:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417186688/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098587.964683:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098587.964687:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098587.964717:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417186624:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098587.964830:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098587.964836:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417186560:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098587.964942:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186688:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766098587.972431:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098587.972445:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098587.972460:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186688:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098587.972473:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417186560:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098587.972493:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417186624:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098588.156627:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098588.732525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098589.181062:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098589.758493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098590.205622:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098590.780477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098591.229531:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098591.813201:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098592.256351:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098592.828849:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098593.084633:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098593.084650:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098593.084654:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098593.084663:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3097 00000100:00100000:2.0:1766098593.084664:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417186752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098593.084667:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098593.084672:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098593.084682:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098593.084697:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417186880/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098593.084722:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098593.084725:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098593.084751:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417186816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098593.084886:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186880:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098593.090001:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417186816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098593.090550:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417186752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098593.090751:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098593.090757:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098593.090762:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417186880:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766098593.277724:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098593.853040:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098594.300904:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098594.876500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098595.324670:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098595.900459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098596.349097:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098596.924508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098597.372651:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098597.953545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098598.204651:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098598.204661:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098598.204666:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098598.204680:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3102 00000100:00080000:0.0:1766098598.204684:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098598.204692:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098598.204708:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098598.204725:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ffe0f9f x1851885417187072/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098598.204788:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187072:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766098598.205221:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098598.205227:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098598.205251:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417187008:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098598.205490:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098598.205496:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417186944:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098598.208694:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098598.208707:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098598.208719:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187072:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098598.209244:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417186944:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098598.209272:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417187008:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098598.396868:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098598.972547:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098599.420825:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098599.996819:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098600.444596:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098601.021434:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098601.468655:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098602.044464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098602.493347:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098603.071057:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098603.324595:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098603.324623:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098603.324628:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098603.324642:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3107 00000100:00080000:0.0:1766098603.324647:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098603.324654:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098603.324669:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098603.324691:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009cc3c5f5 x1851885417187264/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098603.324780:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187264:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098603.324781:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417187136:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098603.324833:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098603.324838:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098603.324857:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417187200:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098603.327996:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098603.328010:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098603.328123:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187264:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098603.328444:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417187136:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098603.332743:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417187200:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098603.520761:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098604.092534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098604.541048:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098605.116579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098605.564630:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098606.140513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098606.588664:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098607.164535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098607.613275:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098608.190118:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098608.444662:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098608.444671:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098608.444675:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098608.444689:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3112 00000100:00080000:0.0:1766098608.444692:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098608.444700:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098608.444713:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098608.444753:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023592e07 x1851885417187456/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098608.444782:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098608.444785:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098608.444826:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417187392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098608.444841:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187456:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098608.445170:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098608.445177:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417187328:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098608.450287:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098608.450301:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098608.450313:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187456:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098608.455377:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417187328:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098608.455406:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417187392:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766098608.637722:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098609.212527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098609.660879:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098610.236530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098610.684547:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098611.261309:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098611.708571:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098612.284880:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098612.736507:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098613.308469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098613.564648:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417187520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098613.566011:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417187584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098613.566822:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098613.566831:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098613.566848:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098613.566855:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3117 00000100:00080000:2.0:1766098613.566859:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098613.566864:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098613.566874:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098613.566894:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1d41706 x1851885417187648/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098613.566963:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187648:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766098613.568187:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098613.568193:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098613.570264:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417187584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098613.570310:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417187520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098613.572972:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098613.573007:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098613.573022:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187648:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098613.756595:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098614.332584:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098614.781943:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098615.356524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098615.804887:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098616.381415:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098616.828548:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098617.405026:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098617.853119:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098618.428493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098618.690998:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098618.691005:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098618.691008:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098618.691018:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3122 00000100:00080000:2.0:1766098618.691021:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098618.691026:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098618.691036:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098618.691050:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417187840/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098618.691072:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098618.691075:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098618.691119:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417187776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098618.691134:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187840:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098618.691378:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417187712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098618.697620:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098618.697637:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098618.697650:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417187840:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098618.700645:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417187712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098618.700709:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417187776:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098618.878041:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098619.452797:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098619.901638:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098620.476680:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098620.926033:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098621.500499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098621.948599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098622.524728:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098622.973319:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098623.549300:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098623.804873:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098623.804881:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098623.804884:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098623.804895:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3127 00000100:00080000:1.0:1766098623.804897:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098623.804904:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098623.804916:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098623.804932:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417188032/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098623.804955:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098623.804958:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098623.805038:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417187904:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098623.805136:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098623.805143:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417187968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098623.805642:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188032:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098623.814003:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417187968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098623.814023:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417187904:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098623.814741:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098623.814753:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098623.814762:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188032:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098623.999318:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098624.578541:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098625.021566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098625.596866:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098626.046646:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098626.620932:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098627.068628:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098627.644480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098628.097279:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098628.668568:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098628.924673:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098628.924680:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098628.924684:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098628.924700:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3132 00000100:00100000:2.0:1766098628.924701:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417188096:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098628.924717:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098628.924723:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098628.924733:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098628.924753:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417188224/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098628.924775:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1766098628.924782:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098628.924790:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417188160:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098628.924791:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098628.924819:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188224:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098628.932819:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417188160:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098628.932837:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417188096:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766098628.933539:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098628.933560:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098628.933565:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188224:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098629.116580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098629.692491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098630.140570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098630.716488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098631.168613:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098631.740799:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098632.189535:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098632.764911:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098633.212582:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098633.789639:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766098634.045549:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417188288:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098634.046918:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098634.046931:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1766098634.046935:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1766098634.046937:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098634.046948:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3138 00000100:00100000:0.0:1766098634.046950:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417188352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098634.046955:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098634.046965:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098634.046979:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098634.047011:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000094c69f2b x1851885417188416/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098634.047028:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098634.047031:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098634.047104:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188416:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098634.053672:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417188288:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098634.058972:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417188352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:3.0:1766098634.059426:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:3.0:1766098634.076057:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098634.076068:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098634.076081:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188416:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098634.238457:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098634.813285:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098635.261266:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098635.836533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098636.284607:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098636.863289:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098637.309320:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098637.884755:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098638.332617:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098638.909081:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098639.164559:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098639.164568:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098639.164573:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098639.164585:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3143 00000100:00080000:1.0:1766098639.164590:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098639.164597:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098639.164610:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098639.164627:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417188608/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098639.164656:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098639.164677:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098639.164730:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417188544:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098639.164851:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098639.164856:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417188480:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098639.165428:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188608:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098639.171590:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417188480:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098639.171615:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417188544:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098639.175408:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098639.175420:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098639.175431:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188608:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098639.362429:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098639.932508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098640.381929:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098640.956480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098641.405925:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098641.980517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098642.429317:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098643.006977:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098643.453331:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098644.028540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098644.284576:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098644.284584:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098644.284588:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098644.284598:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3148 00000100:00080000:1.0:1766098644.284602:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 6s 00000100:00080000:1.0:1766098644.284608:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098644.284617:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098644.284631:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417188800/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098644.284655:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098644.284658:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098644.284688:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417188736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098644.284758:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188800:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098644.284759:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417188672:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098644.289295:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417188736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098644.289388:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417188672:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098644.289418:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098644.289422:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098644.289426:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188800:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098644.477607:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098645.052872:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098645.500804:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098646.080052:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098646.525614:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098647.100688:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098647.549852:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098648.127340:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098648.572539:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098649.150991:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098649.404612:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098649.404619:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098649.404623:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098649.404632:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3153 00000100:00080000:0.0:1766098649.404636:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:0.0:1766098649.404641:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098649.404666:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098649.404683:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417188992/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098649.404707:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098649.404710:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098649.404764:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417188864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098649.404851:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098649.404858:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417188928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098649.406643:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188992:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766098649.415873:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098649.415890:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098649.415898:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417188992:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098649.415958:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417188928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098649.415968:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417188864:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766098649.597071:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098650.172482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098650.621984:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098651.196573:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098651.645753:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098652.221893:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098652.668580:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098653.244470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098653.696060:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098654.268492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098654.525740:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098654.525748:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098654.525751:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098654.525761:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3158 00000100:00080000:1.0:1766098654.525765:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1766098654.525770:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098654.525779:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098654.525791:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417189184/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098654.525813:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098654.525816:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098654.525838:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417189120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098654.525935:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098654.525940:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417189056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098654.526576:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189184:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098654.529924:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417189056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098654.532114:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098654.532124:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098654.532130:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189184:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098654.533676:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417189120:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098654.716594:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098655.295328:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098655.740748:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098656.316474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098656.764591:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098657.340668:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098657.788554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098658.364535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098658.813832:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098659.388474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098659.649919:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098659.649925:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098659.649929:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098659.649939:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3163 00000100:00080000:2.0:1766098659.649943:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766098659.649948:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098659.649957:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098659.649972:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417189376/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098659.649996:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098659.650000:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098659.650063:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189376:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098659.653174:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417189312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098659.653275:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098659.653283:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417189248:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098659.659424:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098659.659435:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098659.659444:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189376:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098659.659596:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417189248:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098659.659609:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417189312:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098659.836689:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098660.412471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098660.860576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098661.438939:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098661.888106:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098662.460485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098662.909689:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098663.484480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098663.932549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098664.509748:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098664.766403:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417189440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098664.766407:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098664.766414:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098664.766418:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098664.766424:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3168 00000100:00080000:3.0:1766098664.766427:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098664.766434:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098664.766445:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098664.766461:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417189568/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098664.766487:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1766098664.766489:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098664.766494:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098664.766495:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417189504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098664.767787:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189568:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098664.786582:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417189504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098664.786632:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417189440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766098664.786997:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098664.787008:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098664.787017:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189568:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098664.958600:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098665.534743:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098665.983219:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098666.557166:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098667.004648:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098667.580509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098668.028639:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098668.606530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098669.052645:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098669.628471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098669.887953:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098669.887961:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098669.887964:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098669.887975:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3173 00000100:00080000:3.0:1766098669.887990:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098669.887996:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098669.888006:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098669.888021:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417189760/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098669.888046:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098669.888050:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098669.888073:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417189696:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098669.888152:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098669.888158:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417189632:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098669.888599:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189760:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098669.890310:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417189632:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098669.890339:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417189696:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098669.891957:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098669.891968:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098669.891978:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189760:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098670.077799:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098670.652536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098671.101574:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098671.676947:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098672.125183:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098672.700501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098673.148650:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098673.724542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098674.173576:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098674.748902:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098675.012942:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417189824:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098675.013089:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098675.013094:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098675.013099:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098675.013105:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3178 00000100:00080000:1.0:1766098675.013108:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098675.013115:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766098675.013124:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417189888:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1766098675.013174:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098675.013194:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f7eb885 x1851885417189952/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098675.013221:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098675.013225:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098675.013733:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189952:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098675.020795:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417189888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098675.021190:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417189824:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098675.022752:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098675.022763:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098675.022772:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417189952:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098675.196675:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098675.772549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098676.220617:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098676.796525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098677.244708:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098677.821283:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098678.268598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098678.844509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098679.296727:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098679.868478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098680.124691:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098680.124700:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098680.124706:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098680.124730:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3184 00000100:00080000:3.0:1766098680.124734:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098680.124742:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098680.124756:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098680.124776:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417190144/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098680.124828:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098680.124833:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098680.124911:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190144:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098680.125808:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417190016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098680.127130:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098680.127144:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417190080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098680.128276:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098680.128292:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098680.128299:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190144:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098680.132756:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417190080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098680.132779:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417190016:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098680.321333:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098680.892475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098681.343084:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098681.916592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098682.365091:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098682.940521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098683.389048:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098683.968660:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098684.412689:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098684.988536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098685.270679:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098685.270688:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098685.270692:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098685.270707:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3189 00000100:00080000:0.0:1766098685.270711:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1766098685.270717:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417190208:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098685.270718:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098685.270733:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098685.270755:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000292c14cb x1851885417190336/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098685.270781:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098685.270785:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098685.270832:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417190272:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098685.271961:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190336:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098685.274805:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417190272:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098685.274928:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417190208:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098685.277435:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098685.277453:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098685.277464:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190336:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098685.436636:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098686.012530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098686.462960:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098687.036516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098687.486904:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098688.062218:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098688.508552:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098689.101264:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098689.532818:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098690.109829:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098690.364543:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098690.364551:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098690.364554:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098690.364586:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3194 00000100:00080000:0.0:1766098690.364589:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098690.364595:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098690.364608:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098690.364629:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417190528/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098690.364637:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417190400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098690.364657:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098690.364662:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098690.364685:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417190464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098690.364831:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190528:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098690.369483:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417190464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098690.369585:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417190400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098690.374443:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098690.374456:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098690.374466:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190528:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098690.556554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098691.132527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098691.580597:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098692.156478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098692.604938:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098693.181619:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098693.629154:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098694.204482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098694.655946:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098695.228475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098695.484583:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098695.484591:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098695.484594:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098695.484605:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3199 00000100:00080000:3.0:1766098695.484608:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098695.484613:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098695.484622:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098695.484636:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417190720/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098695.484663:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098695.484666:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098695.484693:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417190656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098695.484795:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098695.484802:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417190592:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098695.485219:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190720:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098695.492871:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417190656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098695.494898:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417190592:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098695.497744:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098695.497755:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098695.497764:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190720:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098695.676716:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098696.252475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098696.701455:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098697.276573:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098697.724561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098698.303009:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098698.748812:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098699.324500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098699.773006:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098700.348468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098700.604617:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098700.604624:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098700.604627:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098700.604638:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3204 00000100:00080000:3.0:1766098700.604641:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098700.604647:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098700.604657:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098700.604672:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417190912/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098700.604740:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190912:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098700.605027:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098700.605032:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098700.605870:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417190784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098700.605918:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098700.605924:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417190848:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098700.613026:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417190784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098700.614466:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098700.614474:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098700.614481:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417190912:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098700.614989:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417190848:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098700.797178:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098701.372486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098701.820869:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098702.397238:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098702.844646:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098703.420492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098703.869566:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098704.444522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098704.903579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098705.472497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098705.724531:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098705.724538:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098705.724542:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098705.724551:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3209 00000100:00100000:2.0:1766098705.724552:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417190976:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098705.724554:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098705.724570:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098705.724580:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098705.724594:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417191104/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098705.724619:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098705.724622:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098705.724642:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417191040:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098705.724823:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191104:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098705.734599:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417190976:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098705.736888:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417191040:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098705.737407:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098705.737413:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098705.737418:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191104:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098705.918348:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098706.492476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098706.940591:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098707.516491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098707.964881:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098708.540503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098708.991642:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098709.564509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098710.012799:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098710.588503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098710.846227:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098710.846235:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098710.846239:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098710.846249:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3214 00000100:00080000:3.0:1766098710.846252:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098710.846258:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098710.846269:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098710.846286:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417191296/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098710.846310:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098710.846313:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098710.846400:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191296:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098710.846586:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417191232:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098710.846622:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098710.846627:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417191168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098710.853204:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098710.853216:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098710.853226:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191296:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098710.853407:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417191232:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098710.854891:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417191168:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098711.037401:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098711.612473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098712.060602:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098712.638925:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098713.085065:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098713.660455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098714.108578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098714.684463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098715.138617:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098715.710396:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098715.964638:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417191360:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098715.966022:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098715.966033:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417191424:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098715.966754:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098715.966760:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098715.966764:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098715.966770:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3219 00000100:00080000:0.0:1766098715.966773:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098715.966778:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098715.967512:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098715.967542:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2c84c91 x1851885417191488/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098715.967568:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098715.967571:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098715.968125:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191488:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098715.977863:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417191360:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098715.977903:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417191424:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098715.978650:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098715.978655:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098715.978660:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191488:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098716.156740:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098716.732572:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098717.182259:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098717.757432:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098718.204589:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098718.785892:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098719.229276:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098719.805126:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098720.252579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098720.828483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098721.084563:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098721.084584:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098721.084587:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098721.084597:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3225 00000100:00080000:0.0:1766098721.084601:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098721.084606:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098721.084618:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098721.084637:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417191680/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098721.084643:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417191616:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098721.084667:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098721.084671:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098721.084708:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417191552:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098721.084823:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191680:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098721.088932:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417191616:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098721.093592:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417191552:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766098721.094075:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098721.094081:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098721.094086:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191680:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098721.276596:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098721.852495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098722.300585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098722.876475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098723.325563:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098723.900505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098724.349806:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098724.924484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098725.374221:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098725.949193:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098726.204534:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098726.204541:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098726.204545:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098726.204555:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3230 00000100:00080000:3.0:1766098726.204558:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098726.204564:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098726.204573:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098726.204586:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417191872/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098726.204608:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098726.204611:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098726.204687:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191872:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098726.206026:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417191744:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098726.206123:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098726.206130:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417191808:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098726.212794:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098726.212803:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098726.212811:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417191872:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098726.219122:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417191808:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098726.219141:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417191744:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098726.400061:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098726.972497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098727.420620:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098727.996528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098728.446442:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098729.020589:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098729.471872:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098730.044502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098730.493831:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098731.071897:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098731.326299:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417191936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098731.328920:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098731.328946:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417192000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098731.330631:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098731.330643:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098731.330647:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098731.330681:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3235 00000100:00080000:0.0:1766098731.330685:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098731.330692:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098731.330707:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098731.330747:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417192064/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098731.330782:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098731.330785:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098731.330830:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098731.332793:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417192000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098731.332812:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417191936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098731.342524:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098731.342539:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098731.342552:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192064:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098731.516923:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098732.092522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098732.541682:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098733.116676:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098733.566130:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098734.140536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098734.588662:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098735.164524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098735.613430:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098736.188531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098736.444722:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098736.444731:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098736.444736:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098736.444749:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3240 00000100:00080000:3.0:1766098736.444753:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098736.444760:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098736.444773:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098736.444793:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417192256/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098736.444818:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098736.444822:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098736.444857:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417192192:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098736.444858:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192256:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098736.445148:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417192128:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098736.451849:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098736.451865:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098736.451878:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192256:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098736.452016:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417192192:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098736.452050:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417192128:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098736.636614:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098737.216338:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098737.663072:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098738.238033:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098738.685714:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098739.260538:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098739.717143:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098740.284521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098740.732601:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098741.308544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098741.568555:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098741.568565:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098741.568570:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098741.568585:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3245 00000100:00080000:0.0:1766098741.568590:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098741.568598:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098741.568612:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098741.568657:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417192448/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098741.568688:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098741.568692:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098741.568785:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417192320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098741.568915:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098741.568922:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417192384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098741.569091:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098741.572159:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417192384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098741.577910:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098741.577926:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098741.577938:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098741.582166:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417192320:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098741.758432:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098742.332645:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098742.780615:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098743.356846:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098743.804657:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098744.380658:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098744.831618:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098745.404517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098745.858552:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098746.428470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098746.687328:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098746.687336:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098746.687341:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098746.687376:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3250 00000100:00080000:1.0:1766098746.687380:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098746.687388:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098746.687402:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098746.687422:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417192640/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098746.687450:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098746.687454:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098746.687495:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417192576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098746.687630:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098746.687638:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417192512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098746.689102:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192640:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098746.694450:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417192512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098746.694481:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417192576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098746.694762:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098746.694769:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098746.694774:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192640:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098746.877808:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098747.454110:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098747.906285:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098748.477853:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098748.926291:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098749.500828:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098749.948755:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098750.524540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098750.974248:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098751.548470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766098751.806017:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417192704:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098751.807565:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098751.807573:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098751.807578:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098751.807587:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3255 00000100:00080000:0.0:1766098751.807590:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098751.807596:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098751.807610:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098751.807631:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000512a78fc x1851885417192832/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098751.807693:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192832:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766098751.807894:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098751.807898:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098751.808264:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098751.808274:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417192768:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098751.813802:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098751.813815:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098751.813826:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417192832:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098751.813944:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417192768:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098751.813957:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417192704:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098751.996677:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098752.574294:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098753.020624:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098753.596521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098754.044749:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098754.621243:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098755.068713:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098755.644535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098756.093870:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098756.668533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098756.925545:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098756.925573:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098756.925579:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098756.925593:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3260 00000100:00080000:3.0:1766098756.925608:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1766098756.925616:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417192896:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098756.925618:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098756.925634:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098756.925655:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417193024/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098756.925674:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098756.925678:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098756.925727:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417192960:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098756.927252:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193024:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098756.933676:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417192960:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098756.933892:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417192896:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098756.934513:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098756.934522:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098756.934529:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193024:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766098757.117748:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098757.696384:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098758.140642:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098758.716518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098759.164635:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098759.740530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098760.189017:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098760.764524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098761.213737:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098761.788579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098762.047805:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098762.047812:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098762.047816:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098762.047826:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3266 00000100:00080000:2.0:1766098762.047829:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766098762.047834:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098762.047844:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098762.047859:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417193216/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098762.047881:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098762.047884:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098762.047908:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417193152:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098762.048001:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098762.048007:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417193088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098762.048229:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193216:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098762.051334:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417193088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098762.051388:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417193152:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098762.066136:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098762.066146:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098762.066155:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193216:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098762.236555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098762.812480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098763.261345:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098763.836499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098764.285292:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098764.860466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098765.309274:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098765.886089:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098766.332570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098766.908516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098767.164601:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098767.164609:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098767.164612:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098767.164623:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3271 00000100:00080000:1.0:1766098767.164626:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766098767.164632:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098767.164642:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098767.164657:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417193408/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098767.164683:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098767.164686:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098767.164710:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417193344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098767.164852:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417193280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098767.165291:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193408:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098767.190788:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417193344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098767.191243:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417193280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098767.193328:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098767.193343:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098767.193396:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193408:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098767.356646:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098767.936550:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098768.383049:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098768.956543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098769.405208:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098769.980474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098770.429259:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098771.006208:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098771.452575:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098772.028872:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098772.284545:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098772.284552:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098772.284555:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098772.284564:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3276 00000100:00080000:0.0:1766098772.284568:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766098772.284572:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098772.284581:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098772.284594:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417193600/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098772.284614:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098772.284617:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098772.284661:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193600:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098772.284684:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417193472:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098772.284834:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098772.284841:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417193536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098772.290601:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098772.290612:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098772.290619:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193600:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098772.293033:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417193536:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098772.293049:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417193472:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098772.480244:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098773.052470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098773.500797:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098774.076452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098774.524591:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098775.102236:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098775.548572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098776.124471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098776.572522:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098777.149027:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098777.404629:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098777.404637:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098777.404640:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098777.404650:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3281 00000100:00080000:3.0:1766098777.404654:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098777.404659:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098777.404669:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098777.404684:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417193792/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098777.404772:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193792:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098777.405096:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098777.405100:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098777.405122:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417193728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098777.405158:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098777.405164:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417193664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098777.409312:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098777.409323:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098777.409332:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193792:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098777.410604:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417193728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098777.414687:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417193664:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098777.596788:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098778.172480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098778.620774:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098779.198567:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098779.646639:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098780.220528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098780.668762:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098781.244474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098781.693261:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098782.270035:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098782.525421:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098782.525428:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098782.525432:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098782.525445:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3286 00000100:00080000:3.0:1766098782.525449:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098782.525456:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766098782.525460:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417193856:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1766098782.525471:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098782.525494:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417193984/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098782.525510:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098782.525514:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098782.525549:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098782.525557:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417193920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098782.525733:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193984:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098782.528472:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417193920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098782.528492:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417193856:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098782.533600:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098782.533612:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098782.533622:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417193984:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098782.716802:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098783.293052:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098783.740546:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098784.316477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098784.764666:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098785.340473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098785.788563:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098786.365785:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098786.812549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098787.392316:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098787.644537:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098787.644545:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098787.644548:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098787.644558:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3291 00000100:00080000:3.0:1766098787.644561:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098787.644567:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098787.644577:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098787.644592:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417194176/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098787.644617:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098787.644620:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098787.644652:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194112:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098787.644665:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194176:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098787.644747:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098787.644757:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098787.651575:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098787.651598:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194112:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:0.0:1766098787.652694:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:0.0:1766098787.652704:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098787.652709:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098787.652712:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194176:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098787.837291:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098788.412485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098788.860558:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098789.439522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098789.889186:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098790.461888:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098790.910039:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098791.484525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098791.932646:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098792.508438:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098792.764600:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098792.764609:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098792.764614:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098792.764627:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3296 00000100:00080000:3.0:1766098792.764631:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766098792.764638:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098792.764651:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098792.764671:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006a1d607 x1851885417194368/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098792.764757:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194368:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098792.767004:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098792.767016:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098792.767068:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194304:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098792.767180:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098792.767189:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098792.774616:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194304:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098792.776046:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098792.776060:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098792.776070:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194368:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098792.776193:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194240:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098792.957509:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098793.532513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098793.980957:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098794.559489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098795.004551:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098795.586420:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098796.028732:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098796.605691:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098797.052730:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098797.628516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098797.885476:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098797.885483:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098797.885487:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098797.885510:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3301 00000100:00080000:2.0:1766098797.885513:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098797.885519:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098797.885531:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098797.885556:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417194560/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098797.885578:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098797.885581:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098797.885794:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417194432:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098797.885881:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098797.885887:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417194496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098797.886225:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194560:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766098797.892243:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098797.892253:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098797.892262:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194560:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098797.892507:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417194496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098797.892519:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417194432:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098798.081815:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098798.652493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098799.100551:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098799.676473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098800.125234:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098800.700518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098801.148573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098801.724486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098802.177214:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098802.748463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098803.004596:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098803.004603:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098803.004607:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098803.004617:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3306 00000100:00080000:3.0:1766098803.004621:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098803.004625:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098803.004636:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098803.004651:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ebab33a4 x1851885417194752/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098803.004675:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098803.004678:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098803.004708:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098803.004790:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098803.004796:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098803.005227:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194752:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098803.009571:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098803.009592:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098803.010729:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098803.010747:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098803.010753:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194752:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098803.197006:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098803.772703:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098804.220555:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098804.799216:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098805.244651:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098805.820497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098806.273302:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098806.845579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098807.293260:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098807.868491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098808.124549:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098808.124556:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098808.124559:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098808.124568:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3312 00000100:00080000:3.0:1766098808.124571:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098808.124576:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098808.124586:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098808.124600:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417194944/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098808.124624:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098808.124627:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098808.124650:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098808.124739:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098808.124745:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098808.125240:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194944:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098808.132202:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417194816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098808.132227:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417194880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098808.133979:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098808.133988:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098808.133996:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417194944:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098808.316533:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098808.895249:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098809.340529:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098809.916471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098810.364719:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098810.940439:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098811.391007:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098811.964453:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098812.417017:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098812.989040:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098813.244567:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098813.244576:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098813.244593:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098813.244604:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3317 00000100:00080000:0.0:1766098813.244608:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098813.244632:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098813.244642:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098813.244658:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417195136/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098813.244713:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098813.244716:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098813.244747:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417195072:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098813.244836:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098813.244842:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417195008:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098813.245176:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195136:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766098813.251476:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098813.251498:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098813.251507:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195136:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098813.252130:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417195008:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098813.252145:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417195072:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766098813.437008:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098814.012911:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098814.460700:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098815.038131:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098815.484578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098816.060484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098816.508548:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098817.084890:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098817.534929:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098818.108465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098818.372821:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098818.372828:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098818.372831:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098818.372842:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3322 00000100:00080000:3.0:1766098818.372846:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098818.372851:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098818.372863:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098818.372879:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417195328/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098818.372904:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098818.372908:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098818.372931:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417195264:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098818.373005:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098818.373010:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417195200:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098818.373098:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195328:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766098818.376918:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098818.376942:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098818.376952:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195328:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098818.377085:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417195200:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098818.377110:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417195264:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766098818.559783:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098819.132528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098819.580790:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098820.156664:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098820.605827:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098821.180400:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098821.632100:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098822.204520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098822.652798:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098823.229299:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098823.484704:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098823.484712:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098823.484715:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098823.484725:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3327 00000100:00080000:0.0:1766098823.484728:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098823.484733:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098823.484743:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098823.484758:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000030e2f50e x1851885417195520/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098823.484815:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195520:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1766098823.486434:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098823.486442:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098823.486467:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417195456:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098823.486529:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098823.486535:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417195392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098823.487083:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098823.487093:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098823.487102:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195520:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098823.500749:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417195392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098823.500808:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417195456:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766098823.676693:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098824.258760:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098824.701319:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098825.276527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098825.724680:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098826.302065:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098826.749037:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098827.324695:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098827.772774:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098828.348515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098828.604649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098828.604658:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098828.604663:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098828.604675:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3332 00000100:00080000:0.0:1766098828.604679:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098828.604687:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098828.604700:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098828.604719:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2c84c91 x1851885417195712/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098828.604742:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098828.604746:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098828.604777:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417195648:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098828.604844:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195712:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098828.604900:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098828.604910:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417195584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098828.615601:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098828.615611:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098828.615619:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195712:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098828.615989:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417195584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098828.616016:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417195648:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098828.796554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098829.372534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098829.820952:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098830.396529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098830.844604:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098831.420490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098831.868648:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098832.444484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098832.893590:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098833.468464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098833.725557:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417195776:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098833.725583:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098833.725587:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098833.725591:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098833.725596:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3337 00000100:00080000:3.0:1766098833.725598:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098833.725603:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098833.725613:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098833.725626:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52cb2e2 x1851885417195904/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766098833.725641:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098833.725646:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417195840:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098833.725651:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098833.725654:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098833.727300:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195904:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766098833.730458:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417195840:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098833.730493:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417195776:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098833.734994:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098833.735005:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098833.735015:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417195904:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098833.916574:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098834.494239:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098834.940611:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098835.516474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098835.966336:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098836.540499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098836.989427:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098837.567610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098838.012666:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098838.591702:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766098838.845810:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417195968:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098838.849954:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098838.849962:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098838.849965:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098838.849975:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3342 00000100:00080000:3.0:1766098838.849979:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098838.849984:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098838.849995:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098838.850011:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417196096/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098838.850034:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098838.850038:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098838.850117:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196096:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766098838.850676:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417196032:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098838.854040:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417195968:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766098838.856873:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417196032:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098838.860221:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098838.860237:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098838.860270:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196096:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098839.036783:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098839.612531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098840.060743:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098840.637426:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098841.084577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098841.662224:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098842.108579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098842.686603:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098843.132533:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098843.708503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098843.965079:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098843.965086:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098843.965089:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098843.965099:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3347 00000100:00080000:0.0:1766098843.965102:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098843.965108:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098843.965118:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098843.965133:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417196288/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098843.965163:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098843.965167:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098843.965194:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417196224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098843.965282:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098843.965287:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417196160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098843.965741:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766098843.972691:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098843.972703:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098843.972715:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098843.979466:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417196160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098843.979500:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417196224:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098844.157339:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098844.732562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098845.184397:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098845.756540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098846.204880:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098846.786675:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098847.230954:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098847.804532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098848.253261:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098848.828647:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098849.086770:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098849.086779:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098849.086783:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098849.086793:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3353 00000100:00080000:0.0:1766098849.086797:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098849.086802:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098849.086814:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098849.086833:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417196480/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098849.086856:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098849.086859:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098849.086889:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417196416:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098849.086983:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098849.086989:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417196352:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098849.087127:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196480:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098849.090896:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417196416:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098849.094650:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098849.094664:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098849.094675:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196480:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098849.096764:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417196352:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766098849.277180:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098849.856622:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098850.301508:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098850.879741:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098851.324654:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098851.900927:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098852.348710:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098852.924506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098853.372788:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098853.948507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098854.206675:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098854.206684:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098854.206706:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098854.206719:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3358 00000100:00080000:0.0:1766098854.206722:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098854.206729:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098854.206743:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098854.206763:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d602f5b x1851885417196672/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098854.206786:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098854.206790:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098854.206881:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196672:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098854.207148:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417196544:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098854.207224:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098854.207230:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417196608:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098854.213758:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417196544:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098854.217551:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417196608:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766098854.217644:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098854.217649:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098854.217653:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196672:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098854.396646:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098854.972496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098855.420524:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098855.996463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098856.444589:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098857.021870:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098857.471129:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098858.045467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098858.492571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098859.068471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098859.324693:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417196736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098859.327541:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417196736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098859.329511:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417196800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098859.334284:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098859.334295:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098859.334299:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098859.334308:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3363 00000100:00080000:2.0:1766098859.334311:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098859.334317:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098859.335316:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098859.336080:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417196864/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766098859.336157:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766098859.336508:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098859.336513:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098859.336585:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417196800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098859.339567:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098859.339576:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098859.339584:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417196864:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098859.517544:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098860.092501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098860.544051:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098861.118871:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098861.568407:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098862.141235:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098862.588580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098863.164901:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098863.612533:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098864.188718:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098864.444899:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098864.444908:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098864.444912:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098864.444921:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3368 00000100:00080000:1.0:1766098864.444924:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098864.444930:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098864.444940:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098864.444954:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417197056/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098864.444980:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098864.444984:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098864.445009:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417196992:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098864.445028:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197056:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766098864.445101:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098864.445108:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417196928:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098864.452946:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417196992:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098864.454002:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417196928:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098864.454154:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098864.454159:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098864.454163:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197056:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098864.637185:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098865.215012:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098865.660582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098866.238832:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098866.684857:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098867.260494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098867.708557:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098868.284475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098868.733388:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098869.308472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098869.564726:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098869.564733:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098869.564736:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098869.564745:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3373 00000100:00080000:2.0:1766098869.564748:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098869.564753:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098869.564762:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098869.564775:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f8448657 x1851885417197248/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098869.564796:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098869.564799:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098869.564821:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417197184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098869.564903:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098869.564909:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417197120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098869.565006:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197248:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098869.573807:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417197184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098869.574324:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417197120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098869.575620:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098869.575627:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098869.575633:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197248:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098869.756508:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098870.332478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098870.780564:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098871.356469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098871.806225:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098872.380490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098872.831498:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098873.404506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098873.852671:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098874.428475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098874.684561:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098874.684581:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098874.684584:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098874.684593:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3378 00000100:00080000:2.0:1766098874.684596:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098874.684602:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098874.684611:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098874.684625:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000204d6b1d x1851885417197440/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098874.684649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098874.684652:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098874.684902:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417197312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098874.684989:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098874.684995:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417197376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098874.686572:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197440:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098874.689494:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417197312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098874.692620:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417197376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098874.698011:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098874.698026:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098874.698035:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197440:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098874.878211:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098875.452494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098875.900577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098876.476474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098876.932821:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098877.500472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098877.949754:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098878.525966:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098878.975397:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098879.548462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098879.804595:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098879.804603:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098879.804607:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098879.804616:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3383 00000100:00080000:1.0:1766098879.804620:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098879.804626:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098879.804636:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098879.804653:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417197632/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098879.804682:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098879.804685:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098879.804712:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417197568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766098879.804721:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197632:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766098879.804854:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417197504:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766098879.811338:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417197504:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766098879.811470:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417197568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766098879.814406:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098879.814417:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098879.814588:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197632:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766098879.997565:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098880.572433:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098881.021086:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098881.596479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098882.044581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098882.620476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098883.068570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098883.644482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098884.092554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098884.668501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098884.926706:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098884.926713:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098884.926717:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098884.926727:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3388 00000100:00080000:2.0:1766098884.926729:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098884.926746:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098884.926755:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098884.926770:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417197824/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098884.926794:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098884.926797:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098884.927750:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197824:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098884.927953:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417197696:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098884.927988:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098884.927995:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417197760:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098884.947848:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417197760:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098884.947876:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1766098884.947878:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417197696:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098884.947900:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098884.947911:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417197824:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766098885.116589:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098885.694265:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098886.141959:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098886.717052:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098887.166322:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098887.740468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098888.189440:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098888.766469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098889.214151:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098889.790168:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098890.044613:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417197888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098890.045528:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417197952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098890.046710:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098890.046717:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098890.046720:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098890.046727:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3394 00000100:00080000:2.0:1766098890.046730:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098890.046735:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098890.046745:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098890.046762:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b951c02b x1851885417198016/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098890.046816:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198016:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766098890.047276:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098890.047280:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098890.050439:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417197952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098890.050937:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098890.050942:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098890.050947:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198016:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098890.052173:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417197888:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098890.237488:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098890.812477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098891.260660:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098891.839829:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098892.286110:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098892.862281:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098893.308554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098893.888214:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098894.332840:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098894.908485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098895.164590:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098895.164597:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098895.164601:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098895.164612:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3399 00000100:00080000:0.0:1766098895.164615:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098895.164621:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098895.164633:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098895.164651:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417198208/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098895.164693:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098895.164697:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098895.164746:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198208:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098895.165468:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417198080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098895.165527:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098895.165534:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417198144:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098895.169097:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098895.169110:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098895.169120:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198208:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098895.175084:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417198144:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098895.175121:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417198080:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766098895.356572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098895.932642:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098896.381039:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098896.956707:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098897.404567:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098897.980511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098898.429232:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098899.004464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098899.453431:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098900.028516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098900.285206:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098900.285213:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098900.285217:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098900.285228:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3404 00000100:00080000:3.0:1766098900.285232:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098900.285238:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098900.285248:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098900.285263:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbfb1ac7 x1851885417198400/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098900.285282:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098900.285285:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098900.285379:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417198272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098900.285472:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098900.285496:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417198336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098900.286934:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198400:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766098900.298809:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098900.298820:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098900.298833:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198400:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098900.299627:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417198272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098900.299652:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417198336:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098900.476561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098901.052621:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098901.501196:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098902.076475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098902.524578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098903.103447:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098903.550729:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098904.124551:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098904.574381:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098905.148478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098905.409522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098905.409531:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098905.409535:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098905.409562:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3409 00000100:00100000:2.0:1766098905.409566:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417198464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098905.409567:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098905.409576:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098905.409589:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098905.409608:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417198592/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098905.409634:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098905.409637:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098905.409670:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766098905.409678:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417198528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098905.414426:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417198528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098905.416640:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198592:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098905.422410:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417198464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098905.425975:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098905.425999:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098905.426013:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198592:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098905.598973:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098906.172524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098906.622784:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098907.196691:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098907.644702:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098908.220469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098908.668612:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098909.244493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098909.692603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098910.268569:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766098910.524522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098910.524529:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766098910.524532:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766098910.524542:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3414 00000100:00080000:3.0:1766098910.524545:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766098910.524550:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766098910.524559:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766098910.524570:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006a1d607 x1851885417198784/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766098910.524592:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766098910.524595:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098910.524641:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098910.526166:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417198656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098910.528527:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417198720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098910.530616:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098910.530626:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098910.530635:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766098910.532725:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417198656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766098910.534776:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417198720:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766098910.716537:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098911.294930:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098911.740895:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098912.318176:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098912.764632:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098913.341393:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098913.788641:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098914.364515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098914.812588:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098915.391030:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098915.644623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098915.644631:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098915.644635:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098915.644647:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3419 00000100:00080000:0.0:1766098915.644651:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098915.644657:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098915.644867:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098915.644887:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417198976/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098915.644916:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098915.644920:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098915.644941:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198976:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098915.645000:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417198912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098915.645128:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098915.645136:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417198848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098915.647606:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417198848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098915.647626:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417198912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766098915.649257:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766098915.649265:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766098915.649272:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417198976:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098915.837181:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098916.416855:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098916.860554:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098917.436480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098917.887055:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098918.460505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098918.913075:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098919.484504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766098919.932765:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098920.508574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098920.767781:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098920.767789:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098920.767792:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098920.767802:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3424 00000100:00100000:3.0:1766098920.767808:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417199040:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098920.767817:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098920.767822:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098920.767831:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098920.767845:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000204d6b1d x1851885417199168/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098920.767877:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098920.767881:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098920.767901:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417199104:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098920.767918:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199168:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098920.782856:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417199104:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098920.783645:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417199040:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098920.783668:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098920.783684:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098920.783689:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199168:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098920.956532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098921.532476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098921.980560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766098922.560600:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098923.004629:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098923.586192:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098924.028907:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098924.604473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098925.054370:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098925.632128:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098925.884559:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098925.884567:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098925.884572:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098925.884583:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3429 00000100:00080000:0.0:1766098925.884586:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098925.884592:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:1.0:1766098925.884599:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417199232:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1766098925.884602:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098925.884621:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023592e07 x1851885417199360/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098925.884648:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098925.884652:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098925.884675:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417199296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098925.885040:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098925.893678:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417199232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766098925.897406:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098925.897416:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098925.897425:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098925.900889:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417199296:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098926.077090:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098926.652531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098927.101574:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098927.677258:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098928.124545:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098928.700482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098929.148654:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098929.724677:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098930.172500:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098930.748504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098931.004552:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098931.004559:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098931.004562:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098931.004573:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3434 00000100:00100000:0.0:1766098931.004574:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417199488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098931.004576:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098931.004584:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098931.004595:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098931.004611:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417199552/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098931.004635:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098931.004639:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098931.004665:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766098931.004673:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417199424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098931.004767:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199552:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098931.012941:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417199488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098931.019645:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098931.019655:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098931.019663:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199552:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098931.024104:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417199424:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766098931.198478:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098931.772533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098932.224031:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098932.797672:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098933.244887:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098933.822028:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098934.268562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766098934.844521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098935.292628:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098935.869315:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098936.126095:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098936.126102:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098936.126105:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098936.126115:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3440 00000100:00080000:2.0:1766098936.126118:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098936.126123:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098936.126133:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098936.126147:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417199744/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098936.126171:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098936.126189:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098936.126213:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417199680:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098936.126326:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766098936.126333:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417199616:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098936.126667:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199744:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098936.134632:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417199680:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098936.134777:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098936.134781:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098936.134799:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199744:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098936.134910:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417199616:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098936.317968:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098936.893793:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098937.340784:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098937.916785:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098938.364502:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098938.940638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098939.388581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098939.965889:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098940.413383:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098940.988478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098941.245910:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098941.245917:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098941.245921:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098941.245933:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3445 00000100:00080000:1.0:1766098941.245937:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098941.245943:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098941.245952:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098941.245968:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000174f4196 x1851885417199936/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098941.245991:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098941.245995:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766098941.246072:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417199808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098941.246162:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098941.246168:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417199872:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098941.246253:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199936:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098941.252698:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417199872:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098941.252729:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417199808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:3.0:1766098941.252882:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:3.0:1766098941.252892:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098941.252897:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098941.252901:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417199936:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098941.437318:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098942.012691:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098942.460643:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098943.036521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098943.484649:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098944.060481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098944.508593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098945.084516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098945.532532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098946.108474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098946.366130:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098946.366138:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098946.366142:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098946.366152:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3450 00000100:00080000:1.0:1766098946.366156:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766098946.366160:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417200000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098946.366165:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098946.366177:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098946.366193:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000653e2b14 x1851885417200128/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098946.366234:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098946.366237:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098946.366255:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417200064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098946.366496:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200128:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098946.373494:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417200000:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098946.373632:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417200064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098946.373693:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098946.373702:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098946.373710:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200128:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098946.556867:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098947.132517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098947.581519:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098948.156482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098948.604560:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098949.180479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098949.629089:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098950.204437:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098950.652585:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098951.229760:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766098951.484602:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417200192:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098951.485897:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098951.485903:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098951.485906:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098951.485914:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3455 00000100:00080000:2.0:1766098951.485917:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098951.485922:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098951.485931:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098951.485946:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000db8badc0 x1851885417200320/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098951.485971:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098951.485974:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098951.486016:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200320:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098951.486046:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417200256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098951.492113:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417200192:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098951.492868:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417200256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098951.495858:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098951.495873:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098951.495884:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200320:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098951.678886:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098952.252487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098952.701679:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098953.276470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098953.729797:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098954.300475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098954.749141:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098955.324474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098955.773406:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098956.349819:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098956.606198:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098956.606204:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098956.606207:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098956.606217:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3460 00000100:00080000:2.0:1766098956.606220:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098956.606224:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098956.606243:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098956.606258:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417200512/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098956.606283:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098956.606287:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098956.606393:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417200384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098956.606478:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098956.606484:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417200448:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098956.606754:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200512:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098956.615117:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417200448:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098956.615135:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417200384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098956.615199:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098956.615204:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098956.615208:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200512:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098956.797091:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098957.372463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098957.820893:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098958.399037:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098958.845625:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098959.421090:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098959.870826:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098960.444472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098960.892580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098961.468477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098961.724527:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098961.724535:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098961.724539:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098961.724549:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3465 00000100:00080000:2.0:1766098961.724553:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098961.724558:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098961.724578:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098961.724593:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417200704/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766098961.724602:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417200640:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098961.724618:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098961.724622:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098961.724697:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766098961.724704:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417200576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098961.724733:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200704:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766098961.731800:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417200576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098961.731830:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417200640:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098961.733916:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098961.733929:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098961.733940:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200704:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766098961.916914:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098962.492478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098962.941784:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098963.516519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098963.965282:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098964.540481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098964.988580:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098965.564475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098966.013216:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098966.588478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098966.845761:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098966.845769:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098966.845772:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098966.845782:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3470 00000100:00080000:1.0:1766098966.845785:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098966.845790:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098966.845801:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098966.845817:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000653e2b14 x1851885417200896/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098966.845841:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098966.845845:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098966.845870:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417200832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098966.845961:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098966.845966:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417200768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098966.849395:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098966.849541:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417200832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098966.849743:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417200768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766098966.852249:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098966.852265:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098966.852278:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417200896:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098967.036598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098967.612509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098968.060780:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098968.636503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098969.084724:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098969.660626:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098970.108809:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098970.684460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098971.138683:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098971.708701:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098971.964909:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098971.964916:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098971.964919:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098971.964933:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3475 00000100:00080000:2.0:1766098971.964948:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098971.964955:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098971.964964:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098971.964979:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417201088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766098971.965032:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201088:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098971.965043:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417200960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098971.965111:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098971.965116:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417201024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766098971.966388:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098971.966394:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098971.968348:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417200960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766098971.969137:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417201024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098971.969995:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098971.970001:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098971.970007:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201088:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766098972.156848:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098972.732509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098973.184908:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098973.756723:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098974.204796:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098974.780476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098975.229385:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098975.807034:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098976.252565:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098976.829102:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098977.086128:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098977.086134:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098977.086138:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098977.086148:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3481 00000100:00080000:1.0:1766098977.086151:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098977.086157:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098977.086166:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098977.086178:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417201280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098977.086202:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098977.086205:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098977.086229:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417201216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098977.086246:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201280:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766098977.086460:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417201152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766098977.092166:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417201152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098977.092183:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417201216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098977.092280:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098977.092285:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098977.092290:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201280:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766098977.276593:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098977.852510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098978.304021:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098978.876503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098979.335919:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098979.902553:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098980.348646:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098980.924479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098981.373126:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098981.948472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766098982.204553:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098982.204560:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766098982.204564:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766098982.204575:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3486 00000100:00080000:1.0:1766098982.204578:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766098982.204584:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766098982.204594:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766098982.204608:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000174f4196 x1851885417201472/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766098982.204631:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766098982.204635:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098982.204660:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417201408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098982.204756:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766098982.204762:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417201344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766098982.204876:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201472:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766098982.206717:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417201408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766098982.207812:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417201344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766098982.208021:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766098982.208025:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766098982.208029:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201472:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766098982.397385:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098982.972470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098983.420766:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098983.996503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098984.445968:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098985.020497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098985.471773:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098986.045012:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098986.494106:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098987.068490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766098987.324593:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098987.324600:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766098987.324604:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766098987.324614:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3491 00000100:00080000:2.0:1766098987.324617:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766098987.324622:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766098987.324633:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766098987.324648:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b83cf9b1 x1851885417201664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766098987.324677:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766098987.324680:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766098987.324747:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201664:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098987.324963:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417201536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098987.324996:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766098987.325002:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417201600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766098987.335203:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417201536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766098987.335408:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766098987.335413:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766098987.335419:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201664:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766098987.335754:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417201600:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098987.517678:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098988.092529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098988.540599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098989.122541:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098989.564628:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098990.140520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098990.588670:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098991.165809:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098991.612627:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098992.188520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098992.445143:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098992.445153:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098992.445157:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098992.445171:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3496 00000100:00080000:0.0:1766098992.445175:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098992.445200:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098992.445213:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098992.445235:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009cc3c5f5 x1851885417201856/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098992.445264:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098992.445268:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766098992.445323:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201856:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098992.445590:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417201728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098992.445626:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766098992.445631:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417201792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098992.449432:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098992.449448:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098992.449460:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417201856:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766098992.450137:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417201792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766098992.450153:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417201728:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766098992.636843:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098993.212505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098993.662199:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098994.236507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098994.684613:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098995.260630:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098995.708662:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098996.284777:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098996.735840:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098997.308644:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766098997.564605:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098997.564614:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766098997.564618:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766098997.564631:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3501 00000100:00080000:0.0:1766098997.564639:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766098997.564647:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766098997.564660:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766098997.564678:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417202048/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766098997.564706:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766098997.564710:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766098997.564862:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202048:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098997.564887:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417201920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098997.564988:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766098997.565001:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417201984:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766098997.580552:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417201920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766098997.581402:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766098997.581409:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766098997.581415:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202048:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766098997.583256:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417201984:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766098997.756676:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098998.332515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766098998.782314:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766098999.357738:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766098999.804917:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099000.380878:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099000.829846:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099001.404507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099001.852888:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099002.430857:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099002.685885:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099002.685894:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099002.685898:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099002.685912:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3506 00000100:00080000:2.0:1766099002.685915:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099002.685923:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099002.685937:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099002.685956:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417202240/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099002.685987:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099002.685990:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099002.686023:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417202176:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099002.686194:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202240:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099002.686305:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099002.686311:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417202112:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099002.693299:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417202176:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099002.700483:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099002.700498:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099002.700511:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202240:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099002.701169:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417202112:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099002.877701:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099003.453212:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099003.909381:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099004.476640:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099004.924539:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099005.500473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099005.948569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099006.530646:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099006.973859:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099007.548713:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099007.804566:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099007.804573:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099007.804576:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099007.804586:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3511 00000100:00080000:1.0:1766099007.804590:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099007.804595:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099007.804605:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099007.804633:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007d99d644 x1851885417202432/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099007.804659:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099007.804662:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099007.804689:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417202368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099007.804698:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202432:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099007.804791:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099007.817306:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417202368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099007.817464:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099007.817826:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099007.817832:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099007.817837:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202432:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099007.996700:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099008.572473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099009.020558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099009.596523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099010.044951:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099010.620529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099011.068557:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099011.644473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099012.092831:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099012.668772:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099012.924595:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099012.924603:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099012.924606:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099012.924617:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3516 00000100:00080000:0.0:1766099012.924620:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099012.924626:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099012.924636:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099012.924651:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009cc3c5f5 x1851885417202624/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099012.924677:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099012.924681:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099012.924750:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099012.924798:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202624:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099012.924852:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099012.924861:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417202560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099012.930101:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417202560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099012.930118:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202496:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099012.930218:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099012.930224:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099012.930228:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202624:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099013.116545:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099013.692420:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099014.140578:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099014.718068:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099015.164581:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099015.740470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099016.190285:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099016.764498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099017.212943:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099017.798644:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099018.044522:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099018.044530:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099018.044544:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099018.044553:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3522 00000100:00080000:0.0:1766099018.044555:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099018.044560:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099018.044569:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099018.044583:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417202816/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099018.044643:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202816:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099018.045860:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099018.045961:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099018.045965:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099018.045995:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417202752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099018.052671:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099018.052682:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099018.052691:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417202816:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099018.054048:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417202752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099018.054430:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202688:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099018.236571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099018.812621:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099019.262380:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099019.836730:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099020.289811:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099020.860466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099021.308777:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099021.884470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099022.332596:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099022.908464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099023.166441:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099023.166448:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099023.166451:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099023.166461:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3527 00000100:00080000:0.0:1766099023.166464:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099023.166469:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099023.166479:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099023.166494:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000968f8523 x1851885417203008/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099023.166518:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099023.166521:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099023.166553:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417202944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099023.166562:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203008:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099023.166665:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099023.173964:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417202944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099023.174398:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099023.174404:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099023.174408:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203008:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099023.174421:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417202880:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099023.356846:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099023.932486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099024.387145:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099024.956476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099025.404575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099025.980484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099026.428540:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099027.004753:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099027.453176:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099028.028475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099028.284640:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099028.284648:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099028.284652:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099028.284661:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3532 00000100:00080000:2.0:1766099028.284663:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099028.284669:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099028.284679:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099028.284693:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417203200/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099028.284720:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099028.284723:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099028.284748:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417203136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099028.284827:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099028.284833:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417203072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099028.285483:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203200:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766099028.287738:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099028.287748:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099028.287755:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203200:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099028.297325:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417203072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099028.297350:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417203136:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099028.479200:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099029.052530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099029.500554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099030.076471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099030.525964:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099031.100464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099031.548559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099032.124468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099032.573589:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099033.150077:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099033.406315:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099033.406323:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099033.406327:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099033.406339:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3537 00000100:00080000:1.0:1766099033.406343:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099033.406350:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099033.406382:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099033.406399:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417203392/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099033.406414:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099033.406418:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099033.406554:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417203264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099033.406641:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099033.406648:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417203328:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099033.407011:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203392:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099033.420265:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099033.420279:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099033.420290:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203392:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099033.421378:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417203264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099033.421417:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417203328:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099033.596661:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099034.174587:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099034.622814:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099035.196785:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099035.647187:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099036.221113:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099036.670510:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099037.248786:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099037.692580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099038.268905:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766099038.524802:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417203456:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099038.526136:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099038.526145:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099038.526149:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099038.526160:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3542 00000100:00080000:2.0:1766099038.526164:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099038.526171:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099038.526184:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099038.526205:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417203584/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099038.526232:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099038.526235:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099038.526268:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417203520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099038.526398:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203584:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099038.533850:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417203520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099038.533966:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099038.533972:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099038.533977:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203584:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099038.535003:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417203456:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099038.718092:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099039.292558:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099039.741119:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099040.316477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099040.764533:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099041.340541:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099041.790007:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099042.364504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099042.812606:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099043.389753:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099043.646072:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099043.646080:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099043.646083:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099043.646106:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3547 00000100:00100000:3.0:1766099043.646107:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417203648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099043.646110:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099043.646115:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099043.646126:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099043.646142:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417203776/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099043.646168:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099043.646171:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099043.646191:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417203712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099043.648013:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203776:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099043.655079:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417203648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099043.656486:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417203712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099043.656698:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099043.656704:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099043.656708:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203776:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099043.838787:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099044.415689:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099044.860546:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099045.437024:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099045.885722:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099046.460485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099046.908531:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099047.484489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099047.932560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099048.508689:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099048.766271:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099048.766278:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099048.766281:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099048.766291:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3552 00000100:00080000:1.0:1766099048.766294:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099048.766300:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099048.766310:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099048.766342:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417203968/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099048.766397:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099048.766401:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099048.766426:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417203904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099048.766505:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099048.766510:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417203840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099048.766899:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203968:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099048.779082:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099048.779092:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099048.779100:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417203968:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099048.779183:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417203840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099048.779197:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417203904:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099048.956587:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099049.532642:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099049.980596:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099050.556756:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099051.006000:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099051.580801:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099052.029858:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099052.604476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099053.052657:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099053.631122:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099053.885217:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099053.885224:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099053.885227:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099053.885237:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3557 00000100:00080000:2.0:1766099053.885254:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099053.885259:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099053.885270:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099053.885284:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1d41706 x1851885417204160/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099053.885342:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204160:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099053.885675:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417204032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099053.885706:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:2.0:1766099053.886311:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099053.886316:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099053.886521:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417204096:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099053.888925:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417204032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099053.892159:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099053.892169:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099053.892178:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204160:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099053.893084:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417204096:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099054.077139:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099054.652468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099055.101190:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099055.676430:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099056.124625:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099056.700495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099057.149967:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099057.724468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099058.174108:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099058.748511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099059.004948:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099059.004956:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099059.004960:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099059.004971:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3562 00000100:00080000:1.0:1766099059.004974:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099059.004980:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099059.004990:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099059.005019:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417204352/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099059.005046:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099059.005050:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099059.005082:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417204288:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099059.005179:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099059.005186:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417204224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099059.005278:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204352:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099059.013770:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417204288:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099059.014716:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099059.014723:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099059.014728:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204352:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099059.017306:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417204224:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099059.197111:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099059.772456:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099060.220855:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099060.798948:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099061.246480:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099061.820497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099062.269483:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099062.844477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099063.293046:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099063.868475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766099064.124602:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417204416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099064.125308:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099064.125315:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417204480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099064.127686:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417204480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099064.129248:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099064.129258:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099064.129261:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099064.129270:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3568 00000100:00080000:2.0:1766099064.129274:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099064.129280:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099064.129292:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099064.129322:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007b4dad67 x1851885417204544/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099064.129384:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099064.129388:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099064.129487:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204544:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099064.132587:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417204416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099064.135520:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099064.135530:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099064.135539:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204544:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099064.316574:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099064.892620:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099065.341285:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099065.919911:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099066.364541:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099066.940516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099067.392873:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099067.964481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099068.412583:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099068.988470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099069.244522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099069.244530:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099069.244534:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099069.244544:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3573 00000100:00080000:2.0:1766099069.244548:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099069.244554:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099069.244565:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099069.244583:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417204736/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099069.244657:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099069.245969:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417204672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099069.246026:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099069.246034:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417204608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099069.246462:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099069.246466:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099069.252286:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417204608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099069.253057:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099069.253064:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099069.253069:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204736:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099069.265443:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417204672:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099069.437799:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099070.012491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099070.460569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099071.036466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099071.484587:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099072.061880:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099072.508610:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099073.084487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099073.532584:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099074.108474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099074.364744:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099074.364752:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099074.364778:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099074.364787:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3578 00000100:00080000:3.0:1766099074.364790:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099074.364811:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099074.364822:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099074.364838:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417204928/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099074.364860:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099074.364863:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099074.364891:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417204864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099074.364976:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099074.364983:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417204800:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099074.365062:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204928:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099074.371380:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417204800:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099074.381581:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417204864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099074.381712:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099074.381716:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099074.381720:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417204928:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099074.560795:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099075.132582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099075.582062:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099076.156595:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099076.604550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099077.181238:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099077.636110:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099078.204478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099078.652586:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099079.228593:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099079.488463:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099079.488469:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099079.488474:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099079.488494:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3583 00000100:00080000:1.0:1766099079.488497:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099079.488503:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099079.488512:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099079.488525:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417205120/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099079.488551:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099079.488554:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099079.488577:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417205056:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099079.488653:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099079.488658:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417204992:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099079.489475:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205120:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099079.493312:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417204992:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099079.493335:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417205056:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766099079.496557:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099079.496569:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099079.496579:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205120:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099079.677545:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099080.252477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099080.702052:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099081.276465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099081.725777:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099082.301238:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099082.748522:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099083.324497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099083.772574:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099084.350730:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099084.604627:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417205184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099084.605757:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417205248:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099084.606286:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099084.606291:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099084.606294:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099084.606299:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3588 00000100:00080000:3.0:1766099084.606301:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099084.606306:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099084.606314:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099084.606345:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417205312/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099084.606754:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205312:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099084.607179:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099084.607183:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099084.629047:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417205184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099084.632002:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417205248:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099084.633198:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099084.633207:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099084.633216:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205312:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099084.798196:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099085.372529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099085.821161:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099086.396521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099086.845408:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099087.420604:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099087.870153:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099088.444526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099088.893438:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099089.468511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099089.724630:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099089.724638:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099089.724643:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099089.724675:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3593 00000100:00080000:2.0:1766099089.724680:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099089.724687:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099089.724701:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099089.724720:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ba671fa0 x1851885417205504/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099089.724751:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099089.724755:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099089.724857:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205504:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099089.725280:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417205440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099089.725338:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099089.725345:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417205376:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099089.728918:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417205376:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099089.728937:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417205440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099089.729038:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099089.729043:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099089.729047:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205504:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099089.916659:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099090.492475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099090.942671:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099091.516531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099091.964681:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099092.540448:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099092.988679:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099093.564866:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099094.013771:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099094.591619:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099094.844695:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099094.844722:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099094.844727:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099094.844739:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3598 00000100:00080000:1.0:1766099094.844743:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099094.844751:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099094.844764:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099094.844785:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9051765 x1851885417205696/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099094.844810:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099094.844814:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099094.844848:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417205632:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099094.844987:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099094.844992:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417205568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099094.845069:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205696:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099094.848178:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417205568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:02020000:2.0:1766099094.849632:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:2.0:1766099094.849650:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099094.849658:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099094.849663:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205696:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099094.850136:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417205632:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766099095.036655:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099095.612523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099096.060670:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099096.636522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099097.086088:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099097.660532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099098.108599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099098.687235:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099099.132601:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099099.708484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099099.965728:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099099.965745:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099099.965748:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099099.965765:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3603 00000100:00080000:0.0:1766099099.965768:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099099.965786:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099099.965796:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099099.965812:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417205888/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099099.965837:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099099.965840:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099099.966289:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417205760:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099099.966440:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417205824:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099099.966602:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205888:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766099099.971172:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417205824:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766099099.979118:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099099.979130:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099099.979139:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417205888:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099099.979257:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417205760:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766099100.156585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099100.733196:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099101.180558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099101.756488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099102.204575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099102.780471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099103.231312:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099103.804440:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099104.254517:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099104.830659:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099105.084530:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099105.084536:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099105.084540:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099105.084551:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3609 00000100:00080000:0.0:1766099105.084554:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099105.084560:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099105.084571:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099105.084589:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d602f5b x1851885417206080/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099105.084616:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099105.084619:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099105.084657:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417206016:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099105.084771:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099105.084777:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417205952:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099105.084876:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206080:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766099105.092661:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417205952:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099105.092683:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417206016:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766099105.115910:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099105.115921:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099105.115931:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206080:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099105.284091:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099105.852481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099106.300576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099106.876506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099107.324628:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099107.900471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099108.348580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099108.924499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099109.373026:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099109.948604:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099110.204586:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099110.204604:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099110.204607:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099110.204616:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3614 00000100:00080000:3.0:1766099110.204619:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099110.204624:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099110.204633:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099110.204647:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417206272/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099110.204669:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099110.204671:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099110.204760:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206272:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099110.207619:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099110.208699:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099110.208713:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417206208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099110.216662:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099110.216674:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099110.216701:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206272:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099110.218578:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417206208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099110.218613:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206144:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099110.396590:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099110.972479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099111.420560:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099111.996516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099112.445061:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099113.020548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099113.468928:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099114.044505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099114.492568:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099115.068506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099115.324690:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099115.324697:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099115.324701:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099115.324710:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3619 00000100:00080000:3.0:1766099115.324713:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099115.324719:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099115.324729:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099115.324742:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417206464/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099115.324766:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099115.324782:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099115.324816:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206464:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099115.324867:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206336:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099115.324912:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099115.324920:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417206400:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099115.329879:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099115.329890:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099115.329899:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206464:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099115.329997:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206336:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099115.331615:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417206400:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766099115.517828:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099116.092452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099116.540585:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099117.116483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099117.564599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099118.140492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099118.588615:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099119.164522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099119.612687:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099120.188507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099120.444582:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099120.444588:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099120.444592:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099120.444603:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3624 00000100:00080000:3.0:1766099120.444606:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099120.444611:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099120.444621:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099120.444636:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417206656/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099120.444659:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099120.444662:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099120.444762:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417206592:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099120.444853:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099120.444859:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417206528:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099120.445817:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206656:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099120.453989:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417206592:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099120.460858:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417206528:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099120.467853:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099120.467864:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099120.467875:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206656:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099120.646478:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099121.212840:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099121.660708:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099122.238426:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099122.686295:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099123.260520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099123.710537:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099124.284481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099124.733479:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099125.310326:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099125.564552:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099125.564560:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099125.564564:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099125.564574:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3629 00000100:00080000:3.0:1766099125.564577:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099125.564583:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099125.564610:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099125.567229:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417206848/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099125.567265:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099125.567269:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099125.567499:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206720:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099125.567598:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206848:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099125.567665:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099125.567674:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417206784:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099125.571720:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099125.571730:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099125.571739:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417206848:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099125.573004:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206720:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099125.573690:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417206784:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766099125.756543:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099126.332731:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099126.780660:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099127.360535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099127.804691:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099128.380475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099128.828556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099129.406487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099129.854074:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099130.428562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099130.684517:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099130.684524:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099130.684526:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099130.684536:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3634 00000100:00080000:3.0:1766099130.684539:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099130.684544:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099130.684553:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099130.684565:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417207040/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099130.684630:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207040:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099130.685103:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099130.685107:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099130.686194:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417206976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099130.686715:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099130.690630:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099130.690640:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099130.690649:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207040:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099130.695808:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417206912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099130.695835:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417206976:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099130.879279:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099131.452476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099131.900578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099132.476483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099132.924570:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099133.500661:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099133.948550:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099134.524485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099134.973633:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099135.549395:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099135.804689:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099135.804706:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099135.804710:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099135.804735:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3639 00000100:00080000:1.0:1766099135.804738:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099135.804743:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099135.804754:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099135.804770:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417207232/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099135.804799:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099135.804801:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099135.804888:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417207104:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099135.804977:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099135.804983:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417207168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099135.805085:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207232:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766099135.809320:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417207168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766099135.818658:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099135.818668:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099135.818676:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207232:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766099135.819529:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417207104:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766099135.998010:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099136.572474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099137.021159:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099137.596575:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099138.045248:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099138.627269:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099139.071881:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099139.644465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099140.092562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099140.668478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099140.924645:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417207296:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099140.927137:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417207360:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099140.927881:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099140.927887:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099140.927891:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099140.927898:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3644 00000100:00080000:3.0:1766099140.927900:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099140.927906:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099140.927917:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099140.927936:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417207424/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099140.927955:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099140.927959:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099140.928376:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207424:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766099140.930826:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417207296:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099140.930872:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417207360:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099140.938463:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099140.938478:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099140.938490:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207424:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099141.118993:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099141.693734:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099142.141255:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099142.716524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099143.167994:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099143.741739:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099144.191350:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099144.764476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099145.214302:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099145.789903:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099146.044587:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099146.044595:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099146.044611:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099146.044621:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3650 00000100:00080000:0.0:1766099146.044624:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099146.044630:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099146.044639:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099146.044654:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000292c14cb x1851885417207616/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099146.044682:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099146.044686:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099146.044710:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417207552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099146.044790:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099146.044797:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417207488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099146.044959:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207616:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099146.047168:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417207488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099146.047188:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417207552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099146.047298:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099146.047302:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099146.047307:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207616:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099146.236566:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099146.812483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099147.261348:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099147.836435:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099148.284881:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099148.861265:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099149.309178:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099149.886379:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099150.332548:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099150.908564:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766099151.164582:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417207680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099151.167414:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099151.167422:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099151.167426:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099151.167435:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3655 00000100:00080000:1.0:1766099151.167438:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099151.167444:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099151.167453:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099151.167468:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417207808/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099151.167489:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099151.167507:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099151.167521:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207808:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099151.167616:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417207744:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099151.171522:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417207680:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099151.175693:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099151.175703:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099151.175711:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417207808:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766099151.175859:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417207744:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766099151.357382:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099151.932477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099152.381088:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099152.956470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099153.404689:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099153.980495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099154.430650:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099155.004476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099155.452607:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099156.029390:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099156.284622:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099156.284630:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099156.284633:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099156.284644:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3660 00000100:00080000:2.0:1766099156.284647:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099156.284652:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099156.284661:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099156.284677:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b83cf9b1 x1851885417208000/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099156.284717:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099156.284720:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099156.284787:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208000:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099156.285244:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417207872:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099156.285295:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099156.285300:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417207936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099156.294438:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099156.294449:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099156.294459:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208000:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099156.298721:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417207936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099156.299976:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417207872:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099156.476770:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099157.052551:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099157.500622:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099158.076455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099158.524811:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099159.100476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099159.549747:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099160.124463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099160.572563:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099161.148480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099161.404532:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099161.404540:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099161.404544:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099161.404555:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3665 00000100:00080000:2.0:1766099161.404558:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099161.404564:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099161.404575:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099161.404590:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417208192/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099161.404615:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099161.404618:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099161.404794:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417208128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099161.404892:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099161.404899:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417208064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099161.407126:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208192:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099161.410561:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417208064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099161.415518:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417208128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099161.417941:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099161.417951:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099161.417960:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208192:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099161.596639:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099162.172460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099162.623646:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099163.196806:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099163.646458:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099164.220475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099164.670352:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099165.244486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099165.693476:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099166.268469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766099166.525350:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417208256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099166.527302:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099166.527310:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099166.527313:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099166.527322:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3670 00000100:00080000:0.0:1766099166.527326:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099166.527331:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099166.527341:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099166.527373:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417208384/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099166.527398:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099166.527402:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099166.527466:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417208320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099166.527609:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208384:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766099166.538175:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1766099166.538179:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417208256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099166.538184:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099166.538187:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208384:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099166.538315:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417208320:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099166.716644:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099167.294341:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099167.740635:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099168.317687:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099168.765078:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099169.340608:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099169.789308:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099170.364459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099170.812583:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099171.388769:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099171.646586:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417208448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099171.649145:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1766099171.649152:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417208512:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099171.649167:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099171.649170:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099171.649177:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3675 00000100:00080000:3.0:1766099171.649180:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099171.649186:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099171.649197:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099171.649215:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006796bf6d x1851885417208576/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099171.649243:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099171.649247:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099171.653122:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208576:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099171.654414:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417208512:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099171.654822:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417208448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099171.659489:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099171.659501:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099171.659524:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208576:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099171.836936:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099172.412432:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099172.862631:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099173.436479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099173.884588:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099174.461680:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099174.908591:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099175.484733:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099175.933004:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099176.509391:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099176.766162:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099176.766170:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099176.766174:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099176.766187:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3680 00000100:00080000:1.0:1766099176.766191:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099176.766198:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099176.766213:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099176.766232:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417208768/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099176.766254:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099176.766258:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099176.766288:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417208704:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099176.766436:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099176.766444:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417208640:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099176.766841:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208768:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099176.772856:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417208640:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099176.772883:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417208704:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099176.778059:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099176.778076:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099176.778092:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208768:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766099176.958006:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099177.533127:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099177.980799:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099178.558503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099179.004643:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099179.580804:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099180.035537:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099180.604549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099181.052633:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099181.628504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099181.884623:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099181.884632:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099181.884637:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099181.884651:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3685 00000100:00080000:1.0:1766099181.884675:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099181.884682:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099181.884696:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099181.884715:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417208960/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099181.884742:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099181.884746:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099181.884779:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417208896:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099181.884910:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099181.884917:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417208832:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099181.885270:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208960:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766099181.895613:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099181.895627:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099181.895639:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417208960:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099181.901287:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417208896:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099181.904025:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417208832:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766099182.078944:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099182.658188:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099183.100661:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099183.676511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099184.126849:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099184.700483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099185.153992:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099185.728464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099186.172651:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099186.756584:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099187.004928:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099187.004953:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099187.004957:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099187.004967:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3690 00000100:00080000:2.0:1766099187.004970:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099187.004976:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099187.004986:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099187.005003:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417209152/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099187.005035:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099187.005039:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099187.005069:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417209088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099187.005151:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099187.005157:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417209024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099187.005439:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209152:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099187.022590:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099187.022602:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099187.022624:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209152:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099187.022955:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417209024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099187.022970:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417209088:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099187.214723:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099187.772489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099188.220999:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099188.796479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099189.244616:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099189.820470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099190.268819:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099190.844491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099191.292534:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099191.868415:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099192.124653:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099192.124660:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099192.124664:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099192.124674:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3696 00000100:00080000:3.0:1766099192.124677:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099192.124683:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099192.124693:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099192.124709:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004951ed42 x1851885417209344/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099192.124733:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099192.124736:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099192.124759:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417209280:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099192.125020:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209344:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099192.125094:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417209216:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099192.156677:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417209280:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099192.157966:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417209216:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766099192.162154:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099192.162165:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099192.162173:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209344:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766099192.318341:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099192.892516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099193.345245:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099193.916475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099194.365985:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099194.940467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099195.388891:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099195.971388:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099196.416028:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099196.988425:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099197.244558:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099197.244566:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099197.244569:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099197.244593:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3701 00000100:00080000:3.0:1766099197.244597:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099197.244602:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099197.244613:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099197.244629:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417209536/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099197.244654:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099197.244657:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099197.244683:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417209472:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099197.244712:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209536:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766099197.244765:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099197.244773:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417209408:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099197.248056:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417209408:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099197.248083:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417209472:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766099197.250498:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099197.250513:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099197.250525:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209536:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099197.441342:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099198.012511:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099198.461329:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099199.037014:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099199.487594:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099200.061661:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099200.508626:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099201.084458:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099201.532651:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099202.108547:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099202.365744:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099202.365754:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099202.365758:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099202.365772:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3706 00000100:00080000:0.0:1766099202.365805:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099202.365812:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099202.365827:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099202.365848:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c52a4842 x1851885417209728/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099202.365897:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099202.365901:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099202.365995:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417209664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099202.366120:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099202.366126:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417209600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099202.366949:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209728:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099202.372628:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417209600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099202.372658:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099202.372687:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099202.372698:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209728:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099202.374462:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417209664:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099202.558905:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099203.133548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099203.580636:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099204.156651:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099204.604595:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099205.180558:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099205.628541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099206.204966:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099206.652853:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099207.228623:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099207.485059:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099207.485065:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099207.485069:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099207.485079:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3711 00000100:00080000:3.0:1766099207.485082:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099207.485088:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099207.485108:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099207.485123:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417209920/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099207.485180:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209920:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766099207.485440:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417209792:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766099207.485476:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099207.485482:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417209856:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099207.488912:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099207.488923:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766099207.489933:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099207.489942:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099207.489952:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417209920:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766099207.491610:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417209856:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099207.492713:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417209792:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766099207.676549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099208.252528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099208.700598:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099209.276474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099209.725343:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099210.300478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099210.748549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099211.324491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099211.772610:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099212.360046:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099212.611647:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099212.611661:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099212.611665:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099212.611687:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3716 00000100:00080000:1.0:1766099212.611691:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766099212.611695:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417209984:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766099212.611709:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099212.611721:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099212.611738:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417210112/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099212.611769:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099212.611773:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099212.611790:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099212.611798:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417210048:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099212.611891:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210112:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766099212.618120:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417210048:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099212.618751:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1766099212.618753:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417209984:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099212.618757:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099212.618765:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210112:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766099212.796558:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099213.372648:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099213.820565:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099214.396473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099214.844655:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099215.420542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099215.868559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099216.444419:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099216.894681:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099217.471728:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766099217.726730:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417210176:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099217.727795:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417210240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099217.729076:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099217.729083:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099217.729087:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099217.729094:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3721 00000100:00080000:1.0:1766099217.729097:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099217.729102:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099217.729113:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099217.729132:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f763163e x1851885417210304/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099217.729150:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099217.729153:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099217.729378:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210304:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099217.735121:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417210176:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099217.736298:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417210240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099217.749792:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099217.749824:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099217.749835:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210304:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099217.916590:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099218.492537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099218.941112:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099219.519822:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099219.964547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099220.540735:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099220.989805:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099221.564957:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099222.017782:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099222.593933:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099222.845394:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099222.845401:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099222.845415:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099222.845425:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3726 00000100:00080000:1.0:1766099222.845428:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099222.845433:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099222.845443:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099222.845458:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417210496/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099222.845485:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099222.845488:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099222.845510:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417210432:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099222.846511:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417210368:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766099222.846596:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210496:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099222.853673:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417210432:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766099222.853717:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417210368:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766099222.853797:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099222.853801:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099222.853805:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210496:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766099223.037641:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099223.612470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099224.060567:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099224.636492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099225.090152:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099225.660489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099226.108544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099226.684462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099227.136785:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099227.708476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099227.971875:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099227.971882:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099227.971886:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099227.971896:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3731 00000100:00080000:2.0:1766099227.971899:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099227.971905:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099227.971915:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099227.971928:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417210688/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099227.971949:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099227.971952:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099227.971977:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417210624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099227.972056:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099227.972062:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417210560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099227.972306:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210688:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099227.974631:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417210560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099227.974655:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417210624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099227.976484:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099227.976494:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099227.976502:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210688:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099228.157300:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099228.736371:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099229.183533:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099229.756870:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099230.204660:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099230.780506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099231.228629:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099231.804513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099232.255938:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099232.830045:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099233.084533:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099233.084540:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099233.084543:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099233.084562:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3737 00000100:00080000:2.0:1766099233.084565:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099233.084571:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099233.084580:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099233.084595:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000204d6b1d x1851885417210880/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099233.084638:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099233.084641:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099233.084724:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417210752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099233.084812:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099233.084818:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417210816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099233.085183:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210880:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099233.088337:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417210816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099233.088386:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417210752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099233.088514:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099233.088519:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099233.088523:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417210880:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099233.279186:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099233.852478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099234.304304:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099234.878673:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099235.324592:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099235.901447:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099236.349047:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099236.930044:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099237.372618:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099237.962822:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099238.207667:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099238.207675:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099238.207690:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099238.207701:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3742 00000100:00080000:0.0:1766099238.207704:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099238.207710:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099238.207720:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099238.207737:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417211072/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766099238.207803:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211072:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099238.210203:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417210944:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766099238.210398:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099238.210403:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099238.210440:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417211008:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766099238.220712:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099238.220724:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099238.220734:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211072:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766099238.220835:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417210944:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766099238.221171:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417211008:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766099238.396591:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099238.973370:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099239.426011:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099239.996649:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099240.447536:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099241.020469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099241.468686:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099242.044477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099242.492567:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099243.068510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099243.324635:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099243.324643:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099243.324647:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099243.324657:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3747 00000100:00080000:2.0:1766099243.324662:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 6s 00000100:00080000:2.0:1766099243.324666:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099243.324677:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099243.324691:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417211264/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099243.324715:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099243.324719:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099243.324803:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417211136:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099243.324896:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099243.324903:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417211200:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099243.325423:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211264:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099243.334106:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099243.334119:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099243.334130:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211264:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099243.334226:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417211200:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099243.334237:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417211136:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099243.517559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099244.096271:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099244.546576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099245.116554:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099245.564685:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099246.140525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099246.588619:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099247.164562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099247.612631:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099248.188516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099248.447410:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099248.447418:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099248.447422:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099248.447444:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3752 00000100:00080000:2.0:1766099248.447449:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:2.0:1766099248.447457:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099248.447470:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099248.447489:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417211456/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099248.447581:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211456:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099248.450566:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417211392:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099248.450745:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099248.450755:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417211328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099248.454183:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099248.454193:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099248.463500:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417211328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099248.463531:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417211392:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:1.0:1766099248.463552:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:1.0:1766099248.470669:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099248.470680:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099248.470691:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211456:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099248.640141:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099249.215306:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099249.660678:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099250.236542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099250.686458:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099251.260740:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099251.711868:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099252.285302:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099252.732648:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099253.308528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099253.564715:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099253.564742:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099253.564746:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099253.564758:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3757 00000100:00080000:1.0:1766099253.564763:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1766099253.564769:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099253.564781:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099253.564809:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417211648/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099253.564830:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099253.564833:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099253.564864:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417211584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099253.564985:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099253.564989:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417211520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099253.565288:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211648:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766099253.570528:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099253.570540:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099253.570550:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211648:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099253.570647:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417211520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099253.570662:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417211584:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099253.760422:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099254.332507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099254.786578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099255.356482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099255.804518:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099256.383645:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099256.828603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099257.404504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099257.854171:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099258.428475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099258.684587:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099258.684595:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099258.684599:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099258.684610:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3762 00000100:00080000:1.0:1766099258.684614:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099258.684620:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099258.684633:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099258.684666:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f763163e x1851885417211840/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099258.684688:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099258.684692:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099258.684737:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417211776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099258.685007:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211840:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099258.685309:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417211712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099258.687439:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099258.687456:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099258.687464:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417211840:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099258.689207:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417211776:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099258.690183:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417211712:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099258.879830:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099259.452483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099259.900728:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099260.476982:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099260.924529:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099261.500475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099261.949866:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099262.524481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099262.972659:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099263.548816:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099263.804616:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099263.804623:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099263.804626:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099263.804636:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3767 00000100:00080000:2.0:1766099263.804639:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099263.804644:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099263.804654:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099263.804668:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417212032/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099263.804691:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099263.804694:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099263.804777:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417211904:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099263.804868:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099263.804876:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417211968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099263.804963:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212032:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766099263.813740:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099263.813753:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099263.813765:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212032:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099263.814475:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417211968:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099263.814764:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417211904:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099263.996796:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099264.316725:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099264.316760:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417212096:192.168.203.152@tcp:17:kworker/1:2.0 00000100:00100000:1.0:1766099264.321823:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417212096:192.168.203.152@tcp:17:kworker/1:2.0 00000100:00100000:3.0:1766099264.572801:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099265.021232:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099265.596533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099266.044614:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099266.620577:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099267.072081:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099267.649302:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099268.092706:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099268.669207:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099268.924728:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099268.924736:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099268.924742:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099268.924755:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3772 00000100:00080000:1.0:1766099268.924759:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099268.924767:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099268.924779:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099268.924796:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417212288/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099268.924866:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099268.925107:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417212160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099268.925169:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099268.925176:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417212224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099268.925953:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099268.925959:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1766099268.932694:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099268.932709:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099268.932721:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099268.937923:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417212224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099268.937953:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417212160:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099269.117001:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099269.694093:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099270.140570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099270.716478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099271.164633:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099271.740619:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099272.188630:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099272.766304:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099273.212529:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099273.788476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099274.044594:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099274.044601:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099274.044605:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099274.044614:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3778 00000100:00080000:2.0:1766099274.044617:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099274.044623:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099274.044632:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099274.044647:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417212480/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099274.044670:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099274.044673:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099274.044741:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212480:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099274.044873:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417212352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099274.044906:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099274.044924:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417212416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099274.061169:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417212416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099274.061293:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099274.061298:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099274.061302:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212480:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099274.068118:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417212352:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099274.237861:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099274.814405:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099275.260498:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099275.836592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099276.284594:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099276.861388:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099277.308654:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099277.884471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099278.332999:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099278.908474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099279.167434:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099279.167442:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099279.167445:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099279.167456:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3783 00000100:00080000:3.0:1766099279.167460:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099279.167465:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099279.167476:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099279.167491:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417212672/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099279.167515:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099279.167519:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099279.167564:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417212544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099279.167660:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099279.167754:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417212608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099279.169330:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212672:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099279.180743:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417212608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099279.180762:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417212544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099279.181187:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099279.181191:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099279.181196:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212672:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099279.356979:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099279.933610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099280.381191:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099280.957209:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099281.404628:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099281.980475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099282.428849:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099283.004490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099283.452598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099284.028498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099284.286738:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099284.286745:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099284.286748:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099284.286758:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3788 00000100:00080000:1.0:1766099284.286761:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099284.286766:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766099284.286770:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417212736:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1766099284.286775:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099284.286796:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cdd00614 x1851885417212864/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099284.286821:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099284.286824:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099284.286841:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099284.286852:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417212800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099284.314486:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099284.314495:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099284.314504:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417212864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099284.332298:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417212736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099284.335695:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417212800:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099284.482581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099285.052479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099285.500564:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099286.076464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099286.524670:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099287.100466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099287.548531:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099288.124484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099288.572570:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099289.148471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099289.404587:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099289.404594:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099289.404597:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099289.404607:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3793 00000100:00080000:0.0:1766099289.404611:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099289.404628:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099289.404637:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099289.404651:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000030e2f50e x1851885417213056/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099289.404674:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099289.404677:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099289.404700:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417212992:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099289.404783:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099289.404789:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417212928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099289.404904:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213056:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766099289.415142:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099289.415152:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099289.415161:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213056:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099289.415751:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417212928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099289.415765:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417212992:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099289.598877:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099290.172743:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099290.622603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099291.196478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099291.644566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099292.220544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099292.668545:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099293.244515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099293.693126:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099294.270779:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099294.525399:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1766099294.525400:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417213120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099294.525407:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099294.525411:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099294.525417:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3798 00000100:00080000:2.0:1766099294.525421:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099294.525427:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099294.525437:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099294.525452:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1f2cb20 x1851885417213248/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766099294.525485:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099294.525491:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417213184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099294.525515:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213248:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766099294.525784:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099294.525789:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099294.530956:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417213120:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099294.535906:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417213184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099294.548476:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099294.548491:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099294.548503:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213248:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099294.716634:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099295.292549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099295.741185:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099296.316486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099296.768206:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099297.343241:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099297.788744:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099298.365403:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099298.814460:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099299.389000:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099299.644538:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099299.644545:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099299.644548:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099299.644558:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3803 00000100:00080000:0.0:1766099299.644561:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099299.644567:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099299.644576:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099299.644589:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417213440/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099299.644625:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099299.644628:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099299.644650:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417213376:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099299.644733:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099299.644738:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417213312:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099299.644811:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213440:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766099299.649310:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099299.649320:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099299.649330:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213440:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099299.657604:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417213312:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099299.657627:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417213376:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099299.837497:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099300.412516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099300.860571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099301.436518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099301.886081:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099302.460470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099302.908709:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099303.484464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099303.933456:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099304.508486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099304.764612:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417213504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099304.766125:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099304.766133:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099304.766137:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099304.766146:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3808 00000100:00080000:3.0:1766099304.766151:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099304.766158:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099304.766172:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099304.766190:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417213632/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099304.766276:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099304.766753:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099304.766758:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099304.767188:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099304.767198:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417213568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099304.774298:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099304.774312:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099304.774324:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099304.774810:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417213504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099304.776297:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417213568:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099304.957638:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099305.532472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099305.983990:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099306.556455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099307.007530:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099307.580508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099308.028579:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099308.604487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099309.052642:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099309.628533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099309.885638:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099309.885644:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099309.885648:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099309.885657:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3813 00000100:00080000:0.0:1766099309.885660:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099309.885665:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099309.885673:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099309.885685:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417213824/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099309.885709:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099309.885712:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099309.885746:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417213760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099309.885829:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099309.885834:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417213696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099309.886038:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213824:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099309.888679:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417213760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099309.888780:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099309.888785:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099309.888790:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417213824:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099309.891129:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417213696:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099310.076749:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099310.655405:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099311.100584:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099311.676509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099312.124612:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099312.700591:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099313.149926:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099313.724420:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099314.174114:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099314.748483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099315.005618:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099315.005626:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099315.005629:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099315.005640:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3818 00000100:00080000:1.0:1766099315.005644:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099315.005650:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099315.005659:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099315.005685:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417214016/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099315.005707:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099315.005710:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099315.005783:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417213888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099315.005872:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099315.005878:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417213952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099315.006348:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214016:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099315.008601:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417213952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099315.008617:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417213888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099315.009381:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099315.009386:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099315.009390:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214016:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099315.196899:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099315.772474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099316.221593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099316.796478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099317.244569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099317.820474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099318.271524:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099318.844489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099319.293090:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099319.868476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099320.124541:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099320.124548:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099320.124551:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099320.124561:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3824 00000100:00080000:1.0:1766099320.124564:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099320.124569:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099320.124579:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099320.124591:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002600e542 x1851885417214208/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099320.124614:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099320.124617:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099320.124656:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417214144:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099320.124738:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099320.124743:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417214080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099320.124848:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214208:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099320.129979:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417214144:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099320.132133:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417214080:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099320.134594:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099320.134605:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099320.134627:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214208:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099320.316565:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099320.895598:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099321.341200:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099321.916589:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099322.364479:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099322.943095:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099323.388913:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099323.964491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099324.413492:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099324.988474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099325.245573:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099325.245580:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099325.245583:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099325.245593:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3829 00000100:00080000:1.0:1766099325.245596:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099325.245601:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099325.245610:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099325.245623:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000baf3fdb7 x1851885417214400/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099325.245639:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099325.245642:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099325.250960:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417214336:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099325.251075:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099325.251083:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417214272:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099325.251217:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214400:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766099325.257487:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099325.257498:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099325.257506:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214400:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099325.257537:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417214272:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099325.257571:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417214336:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099325.436536:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099326.012599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099326.460573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099327.038062:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099327.484804:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099328.060527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099328.513437:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099329.084467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099329.532786:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099330.110046:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099330.364699:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099330.364712:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099330.364716:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099330.364727:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3834 00000100:00080000:0.0:1766099330.364731:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099330.364736:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099330.364747:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099330.364762:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000024975221 x1851885417214592/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099330.364785:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099330.364788:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099330.364816:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417214528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099330.364919:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099330.364925:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417214464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099330.365004:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214592:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099330.368137:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417214528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099330.380897:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417214464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099330.382467:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099330.382475:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099330.382483:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214592:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099330.559580:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099331.132537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099331.585776:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099332.156486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099332.605054:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099333.180530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099333.628961:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099334.205137:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099334.655886:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099335.228488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099335.484589:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099335.484596:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099335.484599:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099335.484609:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3839 00000100:00080000:2.0:1766099335.484612:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099335.484618:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099335.484627:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099335.484641:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417214784/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099335.484665:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099335.484669:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099335.484701:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417214720:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099335.484779:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099335.484784:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417214656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099335.484892:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214784:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099335.488264:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099335.488275:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099335.488283:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214784:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099335.488819:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417214656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099335.488836:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417214720:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099335.682190:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099336.258916:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099336.703509:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099337.277058:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099337.724575:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099338.300465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099338.748588:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099339.327421:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099339.772560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099340.353095:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099340.606040:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099340.606047:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099340.606068:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099340.606078:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3844 00000100:00080000:2.0:1766099340.606081:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099340.606087:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099340.606096:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099340.606113:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417214976/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766099340.606121:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417214848:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099340.606135:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099340.606138:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099340.606166:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417214912:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099340.606714:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214976:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099340.611925:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417214912:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099340.611973:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417214848:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099340.612070:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099340.612075:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099340.612079:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417214976:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099340.796566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099341.372453:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099341.820587:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099342.396551:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099342.844570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099343.422098:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099343.868580:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099344.444473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099344.892907:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099345.469440:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099345.724490:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099345.724496:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099345.724500:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099345.724509:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3849 00000100:00080000:2.0:1766099345.724512:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099345.724517:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099345.724527:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099345.724540:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417215168/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099345.724612:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215168:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766099345.724700:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099345.724721:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099345.724738:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417215104:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099345.724774:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099345.724780:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417215040:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099345.729106:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099345.729117:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099345.729126:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215168:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099345.731269:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417215040:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099345.732275:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417215104:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099345.916670:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099346.492476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099346.942692:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099347.519803:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099347.966061:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099348.540459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099348.988766:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099349.564480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099350.012574:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099350.592155:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099350.844580:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099350.844586:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099350.844590:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099350.844599:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3854 00000100:00080000:3.0:1766099350.844602:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099350.844607:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099350.844616:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099350.844629:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417215360/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099350.844649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099350.844652:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099350.844687:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417215296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099350.844762:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099350.844767:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417215232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099350.844958:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099350.852385:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417215296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099350.853876:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099350.853884:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099350.853891:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099350.854308:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417215232:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099351.037900:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099351.612475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099352.060565:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099352.636507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099353.084940:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099353.660488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099354.113157:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099354.684491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099355.132688:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099355.710552:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099355.964581:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099355.964590:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099355.964593:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099355.964605:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3859 00000100:00080000:1.0:1766099355.964608:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099355.964613:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099355.964622:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099355.964660:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417215552/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099355.964685:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099355.964688:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099355.964712:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417215424:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099355.964730:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417215488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099355.964860:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215552:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099355.972119:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417215424:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099355.972167:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417215488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099355.972240:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099355.972245:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099355.972249:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215552:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099356.164019:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099356.732479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099357.181698:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099357.756482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099358.207168:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099358.780472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099359.236045:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099359.804498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099360.252575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099360.828858:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099361.084557:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099361.084566:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099361.084570:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099361.084580:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3865 00000100:00080000:1.0:1766099361.084584:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1766099361.084591:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417215616:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099361.084591:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099361.084617:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099361.084634:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007d99d644 x1851885417215744/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099361.084657:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099361.084660:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099361.084692:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417215680:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099361.084798:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215744:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099361.088421:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417215680:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099361.105044:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417215616:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099361.105155:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099361.105159:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099361.105163:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215744:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099361.279490:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099361.852468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099362.300851:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099362.876508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099363.324598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099363.900613:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099364.348624:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099364.924497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099365.372581:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099365.949849:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099366.205023:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417215808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099366.206054:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099366.206060:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099366.206063:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099366.206070:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3870 00000100:00100000:2.0:1766099366.206207:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417215872:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099366.206632:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099366.206640:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099366.206652:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099366.206668:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417215936/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099366.206690:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099366.206693:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099366.206746:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215936:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099366.209027:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417215808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099366.209029:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417215872:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099366.210169:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099366.210175:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099366.210180:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417215936:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099366.396722:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099366.972512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099367.420640:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099367.996486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099368.444558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099369.036667:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099369.470884:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099370.044435:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099370.493286:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099371.068569:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766099371.324647:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099371.325219:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099371.325226:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417216064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099371.326026:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099371.326032:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099371.326036:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099371.326041:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3875 00000100:00080000:2.0:1766099371.326044:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099371.326058:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099371.326068:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099371.326086:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009119b13e x1851885417216128/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099371.326135:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216128:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766099371.326951:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099371.326956:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099371.330106:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417216064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099371.330124:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099371.334622:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099371.334634:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099371.334649:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216128:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099371.517013:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099372.092512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099372.540599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099373.117868:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099373.564595:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099374.140529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099374.588595:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099375.164657:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099375.612769:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099376.190857:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099376.444592:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099376.444599:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099376.444615:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766099376.444622:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417216192:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099376.444623:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3880 00000100:00080000:1.0:1766099376.444628:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099376.444635:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766099376.444715:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099376.444721:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417216256:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1766099376.445615:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099376.445651:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417216320/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099376.445727:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216320:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766099376.446955:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099376.446963:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099376.454025:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417216192:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099376.458784:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417216256:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099376.458828:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099376.458834:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099376.458838:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216320:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099376.643039:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099377.212486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099377.660527:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099378.236825:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099378.701894:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099379.260491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099379.708600:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099380.284482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099380.732558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099381.308521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099381.564623:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099381.564630:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099381.564633:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099381.564642:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3885 00000100:00080000:3.0:1766099381.564645:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099381.564651:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099381.564660:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099381.564674:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417216512/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099381.564697:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099381.564700:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099381.564733:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417216448:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099381.564735:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216512:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099381.564829:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099381.564837:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417216384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099381.567605:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099381.567614:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099381.567621:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216512:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099381.567751:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417216384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099381.567765:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417216448:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099381.756598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099382.332868:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099382.780593:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099383.356519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099383.804559:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099384.380509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099384.831564:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099385.406601:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099385.853290:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099386.428499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099386.684571:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099386.684578:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099386.684581:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099386.684591:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3890 00000100:00080000:3.0:1766099386.684594:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099386.684600:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099386.684610:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099386.684625:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417216704/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099386.684650:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099386.684654:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099386.684671:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099386.684677:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417216640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099386.684754:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216704:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099386.694714:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216576:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099386.695188:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099386.695194:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099386.695200:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216704:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099386.695376:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417216640:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099386.876714:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099387.452729:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099387.901771:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099388.476522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099388.925050:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099389.501240:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099389.949568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099390.524523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099390.975671:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099391.548484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099391.804699:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099391.804706:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099391.804709:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099391.804719:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3895 00000100:00080000:3.0:1766099391.804722:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099391.804728:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099391.804737:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099391.804751:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417216896/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099391.804813:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099391.804950:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099391.804997:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099391.805003:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417216832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099391.805225:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099391.805229:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099391.823636:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417216832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099391.823667:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099391.823673:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099391.823679:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417216896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099391.824525:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216768:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099391.996555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099392.572494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099393.021024:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099393.596483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099394.044564:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099394.620665:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099395.073230:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099395.644530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099396.092532:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099396.668605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099396.924570:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099396.924577:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099396.924581:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766099396.924586:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099396.924588:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3900 00000100:00080000:3.0:1766099396.924590:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099396.924595:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099396.924605:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099396.924619:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417217088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099396.924641:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099396.924656:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099396.924665:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099396.924671:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417217024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099396.924711:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217088:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766099396.943505:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099396.943522:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099396.943535:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217088:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099396.946117:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417217024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099396.946142:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417216960:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099397.117496:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099397.692965:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099398.140586:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099398.716443:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099399.165331:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099399.740749:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099400.188588:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099400.764782:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099401.213837:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099401.788951:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099402.044571:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099402.044579:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099402.044582:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099402.044593:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3906 00000100:00080000:0.0:1766099402.044596:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099402.044601:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099402.044611:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099402.044611:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417217152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099402.044630:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417217280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099402.044656:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099402.044659:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099402.044683:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417217216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099402.044782:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217280:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099402.048878:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417217152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099402.051442:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417217216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:1.0:1766099402.051544:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:1.0:1766099402.051552:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099402.051556:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099402.051560:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217280:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099402.236608:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099402.812531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099403.261769:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099403.836479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099404.284628:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099404.860534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099405.309752:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099405.885050:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099406.336078:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099406.908477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099407.165793:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099407.165799:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099407.165803:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099407.165812:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3911 00000100:00080000:3.0:1766099407.165815:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099407.165820:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099407.165830:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099407.165844:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417217472/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099407.165866:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099407.165869:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099407.165892:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417217408:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099407.165969:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099407.165976:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417217344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099407.166129:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217472:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099407.169093:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417217344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099407.169596:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417217408:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099407.170503:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099407.170510:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099407.170516:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217472:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099407.357078:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099407.932481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099408.381537:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099408.956506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099409.405504:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099409.987393:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099410.429945:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099411.004529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099411.455147:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099412.028528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099412.286599:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099412.286607:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099412.286630:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099412.286642:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3916 00000100:00080000:1.0:1766099412.286647:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099412.286654:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099412.286668:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099412.286685:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417217664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099412.286709:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099412.286712:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099412.286805:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217664:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099412.287150:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417217536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099412.287269:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099412.287280:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417217600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099412.293754:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417217536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099412.296026:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099412.296035:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099412.296044:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217664:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099412.302822:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417217600:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099412.476554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099413.053498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099413.500552:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099414.076498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099414.524669:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099415.101438:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099415.548650:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099416.124478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099416.572535:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099417.148744:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099417.405150:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099417.405156:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099417.405160:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099417.405170:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3921 00000100:00080000:0.0:1766099417.405173:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099417.405208:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099417.405218:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099417.405235:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417217856/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099417.405258:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099417.405262:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099417.405289:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417217792:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099417.405436:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099417.405443:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417217728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099417.405567:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217856:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099417.409108:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099417.409119:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099417.409127:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417217856:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099417.409351:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417217728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099417.409398:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417217792:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099417.596542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099418.172479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099418.621706:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099419.196480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099419.654011:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099420.222428:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099420.669068:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099421.244479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099421.692690:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099422.268577:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099422.524601:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099422.524608:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099422.524611:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099422.524622:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3926 00000100:00080000:3.0:1766099422.524625:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099422.524630:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099422.524640:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099422.524673:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417218048/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099422.524730:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218048:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1766099422.525240:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099422.525244:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099422.525397:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417217920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099422.525442:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099422.525449:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417217984:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099422.530534:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417217984:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099422.530556:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417217920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099422.530639:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099422.530644:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099422.530648:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218048:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099422.716579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099423.295450:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099423.744401:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099424.316732:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099424.766428:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099425.340472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099425.794185:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099426.366203:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099426.812568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099427.388474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099427.644754:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099427.644767:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099427.644771:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099427.644781:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3931 00000100:00080000:2.0:1766099427.644785:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099427.644791:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099427.644801:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099427.644819:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1d41706 x1851885417218240/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099427.644845:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099427.644849:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099427.644872:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417218176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099427.644958:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099427.644964:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218240:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099427.644967:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417218112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099427.658408:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417218112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099427.658435:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417218176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099427.658680:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099427.658685:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099427.658690:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218240:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099427.842504:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099428.412414:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099428.860793:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099429.436480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099429.884569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099430.460494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099430.908592:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099431.484923:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099431.937790:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099432.508486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099432.764619:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417218304:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099432.768569:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099432.768577:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099432.768580:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099432.768591:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3936 00000100:00080000:3.0:1766099432.768594:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099432.768599:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099432.768610:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099432.768627:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000693b1cfa x1851885417218432/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099432.768649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099432.768653:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099432.768686:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218432:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099432.768686:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417218368:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099432.770265:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417218304:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099432.776949:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099432.776959:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099432.776967:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218432:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099432.777742:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417218368:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099432.960382:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099433.532507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099433.984399:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099434.556902:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099435.004585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099435.580479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099436.028593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099436.604474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099437.052705:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099437.629107:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099437.885995:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099437.886002:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099437.886005:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099437.886015:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3941 00000100:00080000:2.0:1766099437.886018:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099437.886023:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099437.886043:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099437.886060:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417218624/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099437.886085:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099437.886088:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099437.886164:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417218496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099437.886250:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099437.886257:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417218560:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099437.886390:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218624:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099437.891506:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417218496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099437.893776:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417218560:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099437.895192:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099437.895216:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099437.895223:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218624:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099438.078938:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099438.652472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099439.100742:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099439.676480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099440.124577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099440.700486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099441.150928:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099441.724490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099442.172742:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099442.748481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099443.004597:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099443.004604:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099443.004608:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099443.004618:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3946 00000100:00080000:1.0:1766099443.004627:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099443.004633:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099443.004643:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099443.004657:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417218816/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099443.004679:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099443.004694:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099443.004777:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417218688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099443.004875:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099443.004886:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417218752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099443.005177:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218816:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099443.011442:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417218688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099443.013466:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417218752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099443.015219:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099443.015228:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099443.015235:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417218816:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099443.198131:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099443.772454:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099444.220517:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099444.798993:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099445.247003:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099445.820478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099446.270722:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099446.846675:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099447.292549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099447.868468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099448.124620:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417218880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099448.124767:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099448.124774:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417218944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099448.130469:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099448.130480:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099448.130484:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099448.130494:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3952 00000100:00080000:3.0:1766099448.130498:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099448.130504:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099448.130518:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099448.130551:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417219008/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099448.130581:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099448.130598:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099448.132196:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219008:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099448.134521:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417218944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099448.134535:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417218880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099448.143209:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099448.143221:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099448.143230:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219008:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099448.318384:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099448.892474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099449.340522:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099449.918915:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099450.364550:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099450.940436:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099451.389074:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099451.964480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099452.413055:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099452.988472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099453.244553:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099453.244561:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099453.244565:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099453.244576:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3957 00000100:00080000:3.0:1766099453.244579:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1766099453.244581:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417219072:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099453.244588:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099453.244600:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099453.244618:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417219200/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099453.244645:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099453.244649:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099453.244667:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417219136:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099453.244779:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219200:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099453.250680:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417219072:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099453.250851:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099453.250868:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099453.250873:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219200:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099453.251455:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417219136:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099453.436538:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099454.013308:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099454.461790:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099455.036476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099455.484495:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099456.060470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099456.508668:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099457.086054:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099457.533799:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099458.108508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099458.365288:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099458.365295:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099458.365299:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099458.365308:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3962 00000100:00080000:0.0:1766099458.365311:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099458.365317:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099458.365325:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099458.365340:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000512a78fc x1851885417219392/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099458.365408:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099458.365412:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099458.365436:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417219328:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099458.365627:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219392:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099458.365635:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417219264:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099458.372945:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417219328:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099458.374197:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099458.374205:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099458.374212:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219392:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099458.376831:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417219264:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099458.556573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099459.132493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099459.580581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099460.157081:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099460.606771:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099461.181044:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099461.628561:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099462.205275:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099462.656044:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099463.228488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099463.486015:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099463.486034:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099463.486038:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099463.486048:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3967 00000100:00080000:2.0:1766099463.486051:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099463.486056:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099463.486066:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099463.486084:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b83cf9b1 x1851885417219584/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099463.486109:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099463.486113:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099463.486131:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417219456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099463.486134:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417219520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099463.486517:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099463.495447:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417219520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099463.495576:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099463.495580:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099463.495584:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099463.496170:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417219456:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099463.677997:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099464.253550:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099464.700526:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099465.276479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099465.724743:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099466.300737:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099466.748580:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099467.324460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099467.772789:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099468.348468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099468.610071:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099468.610079:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099468.610082:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099468.610093:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3972 00000100:00080000:0.0:1766099468.610096:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099468.610101:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099468.610110:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099468.610126:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005ed5ddd4 x1851885417219776/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099468.610150:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099468.610154:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099468.610179:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417219712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099468.610266:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099468.610272:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417219648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099468.610502:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219776:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099468.613417:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417219712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099468.613662:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099468.613668:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099468.613672:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219776:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099468.615529:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417219648:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099468.796572:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099469.372476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099469.820731:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099470.397418:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099470.849471:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099471.420467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099471.870404:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099472.449534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099472.892627:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099473.468473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099473.725069:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099473.725076:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099473.725079:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099473.725090:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3977 00000100:00080000:3.0:1766099473.725093:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099473.725098:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099473.725108:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099473.725124:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417219968/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099473.725144:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099473.725147:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099473.725174:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417219904:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099473.725259:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099473.725265:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417219840:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099473.725522:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219968:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099473.727244:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417219904:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099473.727865:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417219840:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099473.728038:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099473.728043:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099473.728048:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417219968:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099473.916569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099474.492467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099474.940747:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099475.517097:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099475.964576:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099476.540473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099476.992834:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099477.564579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099478.012861:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099478.588956:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099478.845501:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099478.845508:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099478.845511:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099478.845520:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3982 00000100:00080000:2.0:1766099478.845524:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099478.845529:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099478.845538:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099478.845552:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417220160/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099478.845609:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220160:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099478.845884:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417220032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099478.845922:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099478.845928:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417220096:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099478.849952:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099478.849961:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099478.852789:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417220096:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099478.852818:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417220032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099478.853147:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099478.853152:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099478.853157:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220160:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099479.037085:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099479.613560:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099480.060515:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099480.636488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099481.084688:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099481.660476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099482.108534:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099482.684474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099483.132547:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099483.708467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099483.964577:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099483.964586:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099483.964589:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099483.964613:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3987 00000100:00080000:0.0:1766099483.964617:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099483.964622:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099483.964633:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099483.964650:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417220352/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099483.964674:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099483.964677:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099483.964736:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417220288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099483.964830:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099483.964837:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417220224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099483.964931:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099483.967269:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417220224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099483.967385:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099483.967390:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099483.967394:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099483.978824:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417220288:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099484.156583:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099484.732839:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099485.184477:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099485.756481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099486.204472:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099486.780513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099487.228931:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099487.804483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099488.252556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099488.834489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099489.084722:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099489.084730:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099489.084734:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099489.084745:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3993 00000100:00080000:3.0:1766099489.084749:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099489.084756:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766099489.084764:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417220416:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1766099489.084766:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099489.084785:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417220544/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099489.084813:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099489.084817:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099489.084876:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417220480:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099489.086869:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417220416:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099489.088745:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220544:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099489.094448:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417220480:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099489.102316:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099489.102331:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099489.102342:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220544:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099489.277466:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099489.855597:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099490.301967:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099490.892875:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099491.324542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099491.900481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099492.348567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099492.924508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099493.373216:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099493.948501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099494.206909:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099494.206915:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099494.206919:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099494.206928:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 3998 00000100:00080000:0.0:1766099494.206942:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099494.206947:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099494.206957:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099494.206972:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417220736/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099494.207051:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220736:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099494.211493:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099494.211503:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099494.211549:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417220672:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099494.211648:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099494.211656:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417220608:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099494.212386:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099494.212393:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099494.212398:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220736:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099494.219638:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417220608:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099494.224807:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417220672:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099494.396855:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099494.972472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099495.420777:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099495.998830:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099496.444696:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099497.020476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099497.468520:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099498.044477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099498.492543:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099499.068472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099499.324529:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099499.324538:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099499.324541:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099499.324552:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4003 00000100:00080000:0.0:1766099499.324555:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099499.324560:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099499.324581:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099499.324617:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417220928/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099499.324644:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099499.324648:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099499.324673:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417220800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099499.324760:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099499.324767:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417220864:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099499.325089:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220928:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099499.332531:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417220864:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099499.332548:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417220800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099499.332598:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099499.332604:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099499.332609:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417220928:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099499.518961:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099500.093750:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099500.542394:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099501.117180:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099501.564577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099502.140488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099502.588582:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099503.164523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099503.612599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099504.188484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099504.444624:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099504.444630:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099504.444633:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099504.444642:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4008 00000100:00080000:0.0:1766099504.444645:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099504.444651:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099504.444660:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099504.444673:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000024975221 x1851885417221120/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099504.444695:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099504.444711:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099504.444734:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417221056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099504.444737:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221120:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099504.444799:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417220992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099504.453814:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417221056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099504.460642:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417220992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099504.460754:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099504.460775:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099504.460780:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221120:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099504.640512:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099505.212433:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099505.660625:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099506.240918:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099506.686967:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099507.260537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099507.713608:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099508.284547:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099508.736716:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099509.308531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099509.564908:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099509.564917:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099509.564922:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099509.564936:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4013 00000100:00080000:2.0:1766099509.564939:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099509.564947:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099509.564960:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099509.564980:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417221312/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099509.565047:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221312:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766099509.567383:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099509.567396:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099509.567447:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417221248:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099509.567768:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099509.576254:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099509.576270:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099509.576284:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221312:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099509.577798:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099509.577850:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417221248:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099509.757103:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099510.332549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099510.781529:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099511.356819:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099511.805508:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099512.380841:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099512.833874:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099513.404521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099513.856659:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099514.428535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766099514.686524:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417221376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099514.689074:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417221440:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099514.689168:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099514.689197:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099514.689201:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099514.689207:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4018 00000100:00080000:0.0:1766099514.689210:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099514.689216:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099514.689229:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099514.689293:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417221504/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099514.690928:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221504:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099514.695511:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099514.695545:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099514.698863:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417221376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099514.699384:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417221440:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099514.710536:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099514.710550:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099514.710563:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221504:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099514.880958:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099515.452558:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099515.900713:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099516.476766:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099516.929522:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099517.501642:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099517.948726:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099518.524843:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099518.976049:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099519.548483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766099519.806911:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221568:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099519.810677:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099519.810687:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099519.810692:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099519.810724:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4023 00000100:00080000:3.0:1766099519.810728:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099519.810735:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:3.0:1766099519.810753:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417221632:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:2.0:1766099519.813968:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099519.814036:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417221696/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099519.814077:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099519.814081:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099519.814141:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221568:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099519.814251:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221696:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099519.815063:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417221632:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099519.815869:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099519.815894:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099519.815902:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221696:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099519.996696:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099520.572534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099521.021010:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099521.596495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099522.044658:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099522.620515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099523.069493:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099523.644490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099524.092784:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099524.668503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099524.924905:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099524.924914:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099524.924917:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099524.924927:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4028 00000100:00080000:2.0:1766099524.924931:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099524.924937:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099524.924947:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099524.924962:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417221888/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099524.924986:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099524.924990:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099524.925114:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221888:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099524.925919:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099524.926694:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099524.926704:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417221824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099524.930032:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099524.930043:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099524.930051:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417221888:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099524.930068:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417221824:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099524.930488:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221760:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099525.116633:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099525.692516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099526.140592:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099526.716508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099527.164581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099527.740497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099528.189157:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099528.764598:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099529.212542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099529.790263:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099530.044760:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099530.044767:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099530.044770:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099530.044780:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4034 00000100:00080000:3.0:1766099530.044783:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099530.044789:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099530.044799:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099530.044815:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000872f8703 x1851885417222080/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766099530.044883:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222080:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099530.047502:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099530.047583:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099530.047590:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417222016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099530.050920:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099530.050928:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1766099530.053222:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099530.053232:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099530.053241:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222080:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099530.053576:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417222016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099530.053587:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417221952:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099530.238548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099530.813282:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099531.262649:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099531.836477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099532.285912:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099532.860470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099533.308558:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099533.886752:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099534.332552:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099534.908477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099535.165760:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099535.165768:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099535.165771:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1766099535.165787:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417222208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099535.165790:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4039 00000100:00080000:2.0:1766099535.165793:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099535.165798:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099535.165808:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099535.165822:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000db8badc0 x1851885417222272/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099535.165837:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099535.165839:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099535.165877:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417222144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099535.166528:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222272:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099535.169196:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417222144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099535.178831:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099535.178842:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099535.178852:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222272:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099535.178943:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417222208:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099535.357959:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099535.932677:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099536.380581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099536.958127:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099537.404550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099537.980474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099538.428704:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099539.004474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099539.453343:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099540.028592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099540.284586:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099540.284594:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099540.284597:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099540.284607:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4044 00000100:00080000:0.0:1766099540.284610:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099540.284615:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099540.284625:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099540.284657:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000042205631 x1851885417222464/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099540.284682:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099540.284685:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099540.284710:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417222400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099540.284794:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099540.284801:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417222336:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099540.285112:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222464:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099540.296013:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417222336:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099540.296036:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417222400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099540.300954:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099540.300966:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099540.300975:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222464:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099540.477417:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099541.052435:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099541.500877:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099542.076473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099542.526440:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099543.100475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099543.548635:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099544.124544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099544.580801:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099545.148483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099545.404778:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099545.404786:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099545.404789:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099545.404800:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4049 00000100:00080000:3.0:1766099545.404804:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099545.404811:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099545.404821:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099545.404836:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417222656/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099545.404862:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099545.404865:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099545.404886:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417222592:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099545.404965:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222656:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099545.404967:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099545.404978:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417222528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099545.408219:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417222528:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099545.409601:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099545.409610:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099545.409617:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222656:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099545.412299:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417222592:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099545.602897:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099546.172481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099546.620971:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099547.197090:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099547.644993:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099548.222054:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099548.668506:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099549.245072:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099549.692864:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099550.268473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099550.524572:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099550.524578:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099550.524582:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099550.524603:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4054 00000100:00080000:1.0:1766099550.524606:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099550.524611:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099550.524621:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099550.524633:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000403f15ff x1851885417222848/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099550.524653:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099550.524655:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099550.524721:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222848:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099550.524856:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417222720:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099550.524884:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099550.524888:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417222784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099550.530335:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099550.530344:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099550.530363:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417222848:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099550.534130:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417222784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099550.534143:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417222720:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099550.720697:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099551.292499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099551.740564:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099552.316508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099552.764630:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099553.340491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099553.788617:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099554.364487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099554.812601:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099555.388813:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099555.644540:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099555.644548:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099555.644551:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099555.644561:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4059 00000100:00080000:1.0:1766099555.644565:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099555.644571:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099555.644581:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099555.644598:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417223040/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099555.644620:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099555.644623:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099555.644718:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223040:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099555.645789:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417222912:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099555.645867:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099555.645874:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417222976:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:2.0:1766099555.656926:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:2.0:1766099555.656938:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099555.656943:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099555.656947:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223040:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099555.662419:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417222976:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099555.662439:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417222912:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099555.836567:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099556.413477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099556.861023:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099557.436474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099557.886010:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099558.460502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099558.908599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099559.484479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099559.932595:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099560.508485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099560.764591:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099560.764598:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099560.764602:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099560.764612:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4064 00000100:00080000:3.0:1766099560.764615:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099560.764621:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099560.764630:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099560.764644:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417223232/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099560.764668:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099560.764684:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099560.764754:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223232:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099560.765780:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417223104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099560.765847:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099560.765853:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417223168:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099560.767514:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099560.767523:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099560.767542:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223232:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099560.767679:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417223168:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099560.767689:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417223104:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099560.960197:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099561.532640:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099561.980586:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099562.558877:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099563.005267:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099563.580514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099564.028585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099564.604524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099565.060342:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099565.628445:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099565.884571:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099565.884577:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099565.884580:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099565.884589:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4069 00000100:00080000:1.0:1766099565.884592:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099565.884597:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099565.884606:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099565.884617:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417223424/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099565.884637:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099565.884641:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099565.884709:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223424:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099565.885280:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417223296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099565.885337:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099565.885343:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417223360:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099565.893647:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099565.893656:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099565.893666:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223424:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099565.896502:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417223360:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099565.896521:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417223296:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099566.077278:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099566.652477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099567.100576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099567.676474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099568.125513:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099568.700630:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099569.148670:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099569.727560:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099570.177408:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099570.748489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099571.004571:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099571.004578:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099571.004582:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099571.004595:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4074 00000100:00080000:1.0:1766099571.004599:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099571.004605:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766099571.004608:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417223488:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1766099571.004616:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099571.004633:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417223616/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099571.004652:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099571.004655:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099571.004695:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417223552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099571.004858:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223616:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099571.011857:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417223488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099571.011924:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417223552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099571.014509:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099571.014520:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099571.014530:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223616:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099571.196707:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099571.772505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099572.227629:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099572.796479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099573.244608:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099573.820477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099574.268530:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099574.844476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099575.292626:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099575.868477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099576.125018:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099576.125027:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099576.125031:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099576.125042:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4080 00000100:00080000:3.0:1766099576.125045:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099576.125051:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099576.125060:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099576.125074:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417223808/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099576.125148:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223808:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099576.125204:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417223680:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099576.126141:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099576.126146:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099576.126711:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099576.126724:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417223744:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099576.134385:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417223680:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099576.134450:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417223744:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099576.134542:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099576.134548:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099576.134552:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417223808:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099576.319411:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099576.892468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099577.340575:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099577.916502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099578.364581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099578.940965:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099579.389776:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099579.966509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099580.412560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099580.988491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099581.244886:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099581.244893:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099581.244897:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099581.244906:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4085 00000100:00080000:1.0:1766099581.244909:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099581.244914:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099581.244924:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099581.244939:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417224000/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099581.244993:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099581.244997:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099581.245020:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417223936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099581.245110:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099581.245116:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417223872:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099581.245660:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224000:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099581.248028:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417223872:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099581.248121:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099581.248126:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099581.248131:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224000:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099581.248925:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417223936:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099581.437671:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099582.013868:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099582.467711:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099583.036483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099583.488141:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099584.060470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099584.508633:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099585.084464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099585.532579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099586.108434:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099586.364522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099586.364529:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099586.364544:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099586.364554:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4090 00000100:00080000:1.0:1766099586.364557:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099586.364562:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099586.364571:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099586.364584:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417224192/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099586.364646:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224192:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099586.364764:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417224064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099586.364798:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099586.364802:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417224128:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099586.365295:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099586.365299:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099586.372798:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417224128:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099586.372829:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417224064:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099586.374969:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099586.374979:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099586.374988:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224192:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099586.557487:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099587.132583:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099587.584005:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099588.160159:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099588.604693:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099589.181464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099589.628569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099590.204791:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099590.653543:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099591.228559:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099591.484598:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099591.484605:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099591.484608:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099591.484619:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4095 00000100:00080000:3.0:1766099591.484622:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099591.484628:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099591.484637:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099591.484664:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417224384/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099591.484707:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099591.484710:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099591.484876:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417224256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099591.485000:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099591.485007:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417224320:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099591.489693:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417224320:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099591.489716:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417224256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099591.493055:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224384:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766099591.499016:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099591.499026:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099591.499035:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224384:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099591.680180:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099592.252778:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099592.700586:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099593.278248:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099593.724526:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099594.300437:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099594.748837:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099595.324487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099595.773062:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099596.348484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766099596.607722:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417224448:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099596.609010:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417224512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099596.609972:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099596.609979:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099596.609982:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099596.609990:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4100 00000100:00080000:1.0:1766099596.609993:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099596.609998:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099596.610139:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099596.610162:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417224576/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099596.610208:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224576:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766099596.612109:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099596.612118:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099596.616427:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417224512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099596.618592:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099596.618602:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099596.618611:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224576:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099596.619395:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417224448:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099596.802644:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099597.373102:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099597.820585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099598.396527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099598.844581:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099599.420526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099599.869957:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099600.444471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099600.892582:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099601.471610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766099601.724974:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417224640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099601.726026:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099601.726047:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417224704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099601.726781:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099601.726788:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099601.726791:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099601.726796:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4105 00000100:00080000:2.0:1766099601.726800:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099601.726831:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099601.726841:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099601.726864:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417224768/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099601.727178:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099601.727182:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099601.727275:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224768:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766099601.730478:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099601.730491:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099601.730503:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224768:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099601.730635:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417224640:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099601.731286:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417224704:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099601.918080:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099602.493214:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099602.940916:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099603.516520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099603.964637:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099604.543428:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099604.988734:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099605.567237:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099606.012850:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099606.590017:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099606.846043:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099606.846052:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099606.846056:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099606.846072:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4110 00000100:00080000:3.0:1766099606.846076:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099606.846084:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099606.846120:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099606.846139:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417224960/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099606.846218:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224960:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099606.849968:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417224832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099606.850120:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099606.850130:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417224896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099606.852781:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099606.852793:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:3.0:1766099606.855483:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099606.855497:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099606.855510:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417224960:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099606.855711:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417224896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099606.856457:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417224832:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099607.036646:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099607.612496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099608.062208:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099608.636432:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099609.084563:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099609.660479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099610.109474:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099610.684470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099611.132716:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099611.708471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099611.964613:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099611.964622:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099611.964625:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099611.964637:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4115 00000100:00080000:1.0:1766099611.964640:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099611.964645:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099611.964656:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099611.964670:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417225152/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099611.964690:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099611.964693:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099611.964775:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417225024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099611.964870:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099611.964876:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417225088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099611.965121:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225152:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099611.969621:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417225024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099611.972770:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417225088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099611.975007:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099611.975018:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099611.975027:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225152:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099612.156829:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099612.732479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099613.180534:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099613.756490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099614.206208:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099614.780468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099615.228668:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099615.805275:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099616.252786:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099616.828877:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099617.084613:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099617.084620:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099617.084624:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099617.084649:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4121 00000100:00080000:3.0:1766099617.084652:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099617.084659:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099617.084669:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099617.084686:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417225344/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099617.084740:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099617.084744:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099617.085053:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417225216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099617.085145:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099617.085151:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417225280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099617.087118:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225344:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099617.097395:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417225216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099617.101937:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417225280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099617.109533:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099617.109559:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099617.109568:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225344:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099617.276578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099617.853119:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099618.332015:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099618.876506:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099619.389690:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099619.900555:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099620.412638:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099620.933791:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099621.440532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099621.948471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099622.204586:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099622.204592:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099622.204608:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099622.204617:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4126 00000100:00080000:2.0:1766099622.204621:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099622.204625:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099622.204635:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099622.204648:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417225536/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099622.204677:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099622.204680:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099622.204703:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417225472:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099622.204776:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099622.204780:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417225408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099622.205007:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225536:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099622.207855:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417225408:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099622.207880:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417225472:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099622.209568:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099622.209577:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099622.209584:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225536:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099622.461058:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099622.972559:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099623.488869:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099623.996475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099624.511962:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099625.020480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099625.534491:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099626.044463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099626.556746:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099627.071007:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099627.325799:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099627.325806:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099627.325817:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099627.325828:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4131 00000100:00080000:2.0:1766099627.325832:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1766099627.325832:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417225600:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099627.325840:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099627.325850:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099627.325866:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417225728/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099627.325902:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099627.325906:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099627.325916:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099627.325923:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417225664:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099627.325981:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225728:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099627.331096:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417225600:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099627.333383:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099627.333392:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099627.333401:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225728:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099627.340904:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417225664:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099627.580582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099628.092463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099628.605008:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099629.116462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099629.628537:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099630.140468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099630.652557:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099631.164468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099631.677461:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099632.188649:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099632.445675:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099632.445683:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099632.445687:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099632.445697:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4136 00000100:00080000:3.0:1766099632.445700:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099632.445705:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099632.445714:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099632.445729:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417225920/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099632.445741:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099632.445744:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099632.445767:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417225856:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099632.445845:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099632.445852:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417225792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099632.445946:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225920:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099632.485467:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417225792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099632.485491:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417225856:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099632.496235:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099632.496246:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099632.496255:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417225920:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099632.700556:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099633.213738:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099633.724534:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099634.236481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099634.748538:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099635.260625:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099635.772597:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099636.285025:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099636.798246:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099637.308770:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766099637.565820:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417225984:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099637.566837:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417226048:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099637.567724:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099637.567731:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099637.567734:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099637.567740:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4141 00000100:00080000:0.0:1766099637.567742:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099637.567748:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099637.567758:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099637.567776:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417226112/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099637.567803:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099637.567806:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099637.567878:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226112:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099637.577349:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417225984:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099637.579705:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417226048:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099637.580936:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099637.580943:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099637.580948:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226112:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099637.821161:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099638.332468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099638.846161:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099639.356463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099639.868751:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099640.380492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099640.892585:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099641.404491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099641.917635:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099642.429866:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099642.684725:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099642.684733:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099642.684736:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099642.684746:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4146 00000100:00080000:2.0:1766099642.684749:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099642.684755:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099642.684765:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099642.684780:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417226304/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099642.684803:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099642.684806:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099642.685071:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099642.685156:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099642.685163:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417226240:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099642.685244:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226304:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766099642.690249:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099642.690258:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099642.690268:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226304:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099642.690519:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099642.690542:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417226240:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099642.941167:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099643.454809:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099643.965334:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099644.476551:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099644.988772:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099645.500464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099646.012626:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099646.524514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099647.037560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099647.548730:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099647.804584:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099647.804593:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099647.804596:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099647.804607:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4151 00000100:00080000:2.0:1766099647.804610:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099647.804615:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099647.804625:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099647.804638:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417226496/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099647.804682:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099647.804685:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099647.804779:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226496:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099647.805262:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099647.805309:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099647.805316:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417226432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099647.809737:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417226432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099647.809760:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099647.809856:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099647.809867:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099647.809875:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226496:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099648.061446:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099648.574912:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099649.084582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099649.596472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099650.108791:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099650.620520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099651.133099:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099651.645578:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099652.156574:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099652.668484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099652.925083:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099652.925090:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099652.925094:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099652.925103:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4156 00000100:00080000:3.0:1766099652.925107:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099652.925112:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099652.925122:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099652.925136:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417226688/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099652.925161:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099652.925164:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099652.925192:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417226624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099652.925199:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226688:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099652.925300:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226560:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099652.929079:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099652.929090:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099652.929099:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226688:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099652.929133:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417226624:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099652.929181:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226560:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099653.184110:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099653.694294:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099654.204560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099654.716493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099655.228572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099655.740481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099656.252554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099656.764488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099657.278931:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099657.788483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099658.045802:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099658.045810:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099658.045813:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099658.045824:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4162 00000100:00080000:3.0:1766099658.045827:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099658.045833:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:2.0:1766099658.045838:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226752:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1766099658.045858:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099658.045877:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001161032f x1851885417226880/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099658.045902:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099658.045906:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099658.045928:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099658.045936:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417226816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099658.046064:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226880:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099658.048626:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099658.048637:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099658.048645:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417226880:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099658.048731:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417226816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099658.048742:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417226752:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099658.301857:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099658.812487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099659.324880:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099659.836461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099660.349661:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099660.860520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099661.372585:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099661.886053:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099662.403603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099662.909454:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099663.165935:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099663.165942:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099663.165945:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099663.165968:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4167 00000100:00080000:2.0:1766099663.165971:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099663.165977:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099663.165987:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099663.166001:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f8448657 x1851885417227072/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099663.166024:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099663.166027:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099663.166102:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227072:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099663.166580:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417227008:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099663.166625:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099663.166632:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417226944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099663.169561:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417226944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099663.169946:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099663.169951:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099663.169955:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227072:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099663.170140:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417227008:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099663.421019:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099663.932516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099664.444577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099664.956915:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099665.469719:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099665.980874:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099666.492558:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099667.004513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099667.517644:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099668.028435:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099668.284551:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099668.284558:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099668.284562:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099668.284573:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4172 00000100:00080000:2.0:1766099668.284576:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099668.284582:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099668.284591:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099668.284607:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417227264/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099668.284628:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099668.284631:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099668.284682:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227264:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099668.286322:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417227136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099668.288546:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099668.288557:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099668.288565:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227264:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099668.290942:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099668.290965:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417227200:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099668.291216:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417227136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099668.301426:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417227200:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099668.545290:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099669.052509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099669.564620:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099670.077022:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099670.588899:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099671.100472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099671.612745:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099672.124479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099672.637249:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099673.149900:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099673.405466:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099673.405473:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099673.405478:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099673.405487:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4177 00000100:00080000:3.0:1766099673.405504:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099673.405510:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099673.405520:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099673.405535:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417227456/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099673.405601:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227456:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099673.406085:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417227328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099673.406123:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099673.406130:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417227392:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099673.406912:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099673.406918:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766099673.419020:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099673.419032:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099673.419042:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227456:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099673.419590:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417227392:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099673.419602:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417227328:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099673.660568:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099674.172826:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099674.685732:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099675.196478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099675.714672:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099676.220478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099676.732569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099677.244471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099677.763394:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099678.268485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099678.524680:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099678.524687:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099678.524691:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099678.524700:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4182 00000100:00080000:2.0:1766099678.524703:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099678.524709:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099678.524720:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099678.524735:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417227648/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099678.524771:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099678.524774:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099678.524922:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417227584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099678.525005:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099678.525012:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417227520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099678.525960:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227648:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099678.535559:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417227520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099678.536625:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099678.536632:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099678.536638:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227648:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099678.537247:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417227584:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099678.780537:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099679.292470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099679.804578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099680.316481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099680.828556:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099681.340484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099681.853055:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099682.364493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099682.876599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099683.388521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099683.644588:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099683.644596:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099683.644600:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099683.644609:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4187 00000100:00080000:3.0:1766099683.644612:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099683.644618:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099683.644628:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099683.644643:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417227840/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099683.644662:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099683.644665:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099683.644691:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417227776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099683.644784:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099683.644790:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417227712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099683.645389:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227840:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099683.648928:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417227712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099683.648951:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417227776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099683.649045:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099683.649051:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099683.649055:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417227840:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099683.901156:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099684.412465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099684.924562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099685.436464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099685.948571:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099686.460609:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099686.973927:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099687.484495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099687.996582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099688.508478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099688.764531:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099688.764538:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099688.764542:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099688.764551:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4192 00000100:00080000:3.0:1766099688.764554:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099688.764559:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099688.764568:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099688.764580:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417228032/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099688.764601:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099688.764604:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099688.764674:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228032:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099688.764707:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417227904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099688.764773:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099688.764779:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417227968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099688.773739:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417227904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099688.774305:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417227968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099688.774912:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099688.774918:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099688.774923:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228032:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099689.022328:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099689.533067:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099690.045099:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099690.556587:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099691.068744:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099691.580480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099692.092577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099692.604466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099693.116690:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099693.629387:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099693.884533:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099693.884540:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099693.884543:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099693.884553:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4197 00000100:00080000:3.0:1766099693.884556:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099693.884561:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099693.884571:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099693.884585:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417228224/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099693.884629:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099693.884633:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099693.884657:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417228160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099693.884745:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099693.884751:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417228096:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099693.884950:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228224:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099693.892791:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417228096:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099693.898080:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1766099693.898085:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417228160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099693.898088:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099693.898091:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228224:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099694.141372:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099694.655294:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099695.165377:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099695.676470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099696.189214:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099696.700509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099697.212957:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099697.724940:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099698.236572:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099698.748499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099699.005045:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099699.005055:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099699.005058:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099699.005069:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4202 00000100:00080000:2.0:1766099699.005072:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099699.005077:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099699.005090:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099699.005137:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417228416/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099699.005162:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099699.005165:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099699.005193:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417228352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099699.005280:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099699.005299:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417228288:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099699.007708:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228416:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766099699.012768:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1766099699.012776:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417228288:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099699.012779:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099699.012786:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228416:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099699.012800:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417228352:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099699.264957:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099699.772488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099700.287496:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099700.796473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099701.308575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099701.821275:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099702.333424:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099702.848011:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099703.356555:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099703.870191:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099704.125964:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099704.125971:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099704.125974:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099704.125984:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4208 00000100:00080000:3.0:1766099704.125987:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099704.125992:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099704.126001:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099704.126014:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417228608/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099704.126036:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099704.126039:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099704.126723:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417228480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099704.126826:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099704.126834:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417228544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099704.127276:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228608:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766099704.129411:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099704.129421:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099704.129429:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228608:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099704.131143:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417228544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099704.131158:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417228480:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099704.382790:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099704.892481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099705.406270:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099705.919715:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099706.431428:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099706.940684:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099707.453642:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099707.964449:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099708.476571:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099708.988481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099709.246522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099709.246529:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099709.246533:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099709.246543:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4213 00000100:00080000:2.0:1766099709.246546:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099709.246551:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099709.246563:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099709.246578:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417228800/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099709.246603:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099709.246620:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099709.246647:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228800:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099709.246647:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417228736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099709.246734:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099709.246740:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417228672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099709.249489:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417228736:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099709.256922:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417228672:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1766099709.272141:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:0.0:1766099709.272154:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099709.272159:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099709.272163:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228800:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099709.500571:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099710.012430:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099710.524584:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099711.036557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099711.548595:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099712.060495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099712.575228:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099713.092403:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099713.596595:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099714.108825:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099714.366127:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099714.366134:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099714.366138:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099714.366147:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4218 00000100:00080000:3.0:1766099714.366151:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099714.366156:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099714.366167:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099714.366181:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417228992/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099714.366205:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099714.366209:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099714.366247:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228992:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099714.366387:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417228864:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099714.366418:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099714.366422:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417228928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099714.372316:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417228928:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099714.372559:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099714.372563:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099714.372580:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417228992:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099714.372713:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417228864:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099714.620570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099715.135599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099715.644592:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099716.156487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099716.668661:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099717.180479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099717.693299:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099718.204432:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099718.717094:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099719.228533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099719.486270:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099719.486279:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099719.486284:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099719.486297:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4223 00000100:00080000:2.0:1766099719.486301:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099719.486308:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099719.486322:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099719.486341:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417229184/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099719.486435:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229184:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099719.486737:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417229056:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099719.486783:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099719.486789:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417229120:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099719.487593:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099719.487599:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099719.494145:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417229120:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099719.494186:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417229056:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099719.494418:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099719.494424:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099719.494428:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229184:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099719.745210:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099720.254424:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099720.769017:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099721.276483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099721.788539:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099722.300473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099722.814674:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099723.324594:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099723.838468:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099724.350579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099724.604575:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099724.604584:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099724.604587:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099724.604597:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4228 00000100:00080000:2.0:1766099724.604601:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099724.604606:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099724.604616:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099724.604631:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417229376/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099724.604654:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099724.604657:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099724.604684:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417229312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099724.604695:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229376:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099724.604768:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099724.604775:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417229248:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099724.614202:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417229312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099724.617329:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417229248:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099724.627407:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099724.627418:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099724.627427:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229376:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766099724.861178:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099725.372856:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099725.884525:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099726.396490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099726.908769:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099727.420530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099727.936421:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099728.444914:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099728.956575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099729.468495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099729.727004:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099729.727012:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099729.727016:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099729.727025:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4233 00000100:00080000:0.0:1766099729.727028:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099729.727045:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099729.727055:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099729.727071:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417229568/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099729.727097:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099729.727099:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099729.727166:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417229440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099729.727255:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099729.727261:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417229504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099729.727350:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229568:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099729.730111:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417229440:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099729.731872:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099729.731891:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099729.731899:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229568:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099729.732768:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417229504:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099729.981546:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099730.492489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099731.004569:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099731.517231:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099732.028581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099732.540484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099733.052480:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099733.564588:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099734.076673:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099734.590239:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099734.844536:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099734.844544:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099734.844547:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099734.844557:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4238 00000100:00080000:0.0:1766099734.844560:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099734.844566:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099734.844576:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099734.844591:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000292c14cb x1851885417229760/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099734.844615:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099734.844619:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099734.844705:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417229696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099734.844798:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099734.844805:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417229632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099734.844926:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229760:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099734.848307:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417229696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099734.849286:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417229632:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099734.849941:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099734.850138:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099734.850144:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229760:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099735.101062:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099735.612495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099736.124554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099736.638423:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099737.150470:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099737.660493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099738.172568:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099738.684487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099739.199134:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099739.708466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099739.964850:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099739.964857:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099739.964860:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099739.964869:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4243 00000100:00080000:3.0:1766099739.964872:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099739.964877:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099739.964887:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099739.964901:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417229952/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099739.964925:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099739.964928:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099739.965010:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417229824:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099739.965099:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099739.965122:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417229888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099739.965657:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229952:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099739.972298:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417229888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099739.973323:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099739.973342:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099739.973348:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417229952:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099739.974319:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417229824:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099740.223698:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099740.732478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099741.244578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099741.756475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099742.272289:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099742.780490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099743.292579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099743.807342:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099744.321529:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099744.828463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099745.084709:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099745.084715:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099745.084719:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099745.084729:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4249 00000100:00080000:0.0:1766099745.084733:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099745.084738:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099745.084748:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099745.084762:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417230144/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099745.084800:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099745.084803:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099745.084873:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417230016:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099745.084952:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099745.084957:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417230080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099745.085044:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230144:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099745.088337:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417230080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099745.088388:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417230016:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099745.091759:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099745.091779:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099745.091787:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230144:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099745.341293:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099745.853208:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099746.364681:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099746.876633:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099747.388590:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099747.900466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099748.414627:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099748.924805:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099749.436579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099749.948516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099750.204779:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099750.204786:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099750.204789:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099750.204799:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4254 00000100:00080000:0.0:1766099750.204802:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099750.204807:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099750.204817:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099750.204830:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009cc3c5f5 x1851885417230336/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099750.204856:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099750.204859:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099750.205165:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230336:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099750.205432:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417230208:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099750.205478:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099750.205484:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417230272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099750.222125:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417230272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099750.222169:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099750.222179:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099750.222189:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230336:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099750.223569:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417230208:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099750.461829:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099750.972586:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099751.487833:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099752.000167:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099752.509226:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099753.024125:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099753.532518:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099754.044505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099754.556831:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099755.070298:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099755.326421:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099755.326427:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099755.326431:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099755.326440:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4259 00000100:00080000:0.0:1766099755.326443:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099755.326448:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099755.326457:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099755.326471:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009cc3c5f5 x1851885417230528/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099755.326491:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099755.326494:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099755.326575:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417230464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099755.326678:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099755.326685:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417230400:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099755.327126:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230528:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766099755.331497:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099755.331520:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099755.331528:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230528:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099755.331737:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417230464:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099755.331753:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417230400:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099755.582168:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099756.092498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099756.605071:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099757.116472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099757.630225:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099758.140780:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099758.660759:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099759.164558:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099759.678539:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099760.188530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099760.444614:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099760.444621:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099760.444624:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099760.444635:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4264 00000100:00080000:3.0:1766099760.444638:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099760.444644:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099760.444664:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099760.444680:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417230720/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099760.444705:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099760.444708:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099760.444736:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417230656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099760.444822:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099760.444829:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417230592:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099760.445102:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230720:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099760.447975:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417230592:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099760.448009:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417230656:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099760.453170:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099760.453180:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099760.453189:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230720:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099760.703038:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099761.212471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099761.725882:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099762.238031:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099762.748602:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099763.260472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099763.772679:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099764.286748:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099764.798443:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099765.315931:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099765.564596:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099765.564603:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099765.564607:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099765.564647:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4269 00000100:00080000:2.0:1766099765.564650:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099765.564656:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099765.564665:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099765.564679:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b83cf9b1 x1851885417230912/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099765.564701:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099765.564704:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099765.564731:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417230848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099765.564831:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099765.564836:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417230784:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099765.564917:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230912:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099765.567651:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417230848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099765.571191:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417230784:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099765.577139:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099765.577149:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099765.577158:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417230912:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099765.822080:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099766.333153:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099766.844592:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099767.357324:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099767.871829:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099768.385026:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099768.892606:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099769.405174:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099769.916621:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099770.428462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099770.684544:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099770.684552:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099770.684555:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099770.684565:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4274 00000100:00080000:2.0:1766099770.684568:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099770.684574:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099770.684598:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099770.684615:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ba671fa0 x1851885417231104/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099770.684639:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099770.684642:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099770.684666:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417231040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099770.684743:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099770.684749:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417230976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099770.685170:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231104:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766099770.693670:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1766099770.693675:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417230976:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099770.693679:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099770.693689:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231104:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099770.693703:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417231040:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766099770.940684:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099771.452545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099771.964617:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099772.476532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099772.990239:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099773.500487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099774.013419:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099774.524469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099775.036967:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099775.548481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099775.805420:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099775.805427:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099775.805430:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099775.805440:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4279 00000100:00080000:1.0:1766099775.805444:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099775.805449:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099775.805458:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099775.805490:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417231296/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099775.805512:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099775.805515:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099775.805551:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417231232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099775.805820:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417231168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099775.806275:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231296:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766099775.811343:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099775.811379:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099775.811389:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231296:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099775.811616:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417231168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099775.813421:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417231232:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099776.060993:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099776.572483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099777.084656:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099777.596493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099778.110905:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099778.625806:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099779.132609:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099779.644467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099780.156745:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099780.668446:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099780.924545:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099780.924552:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099780.924556:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099780.924565:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4284 00000100:00080000:2.0:1766099780.924569:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099780.924574:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099780.924584:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099780.924599:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417231488/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099780.924632:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099780.924636:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099780.924662:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417231424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099780.924671:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231488:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099780.924771:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417231360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099780.934107:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417231360:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099780.934154:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417231424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099780.934496:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099780.934501:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099780.934515:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231488:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099781.180972:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099781.692497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099782.206436:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099782.716477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099783.228596:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099783.740467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099784.252568:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099784.764517:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099785.276793:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099785.788543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099786.044820:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099786.044828:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099786.044831:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099786.044842:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4290 00000100:00080000:0.0:1766099786.044846:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099786.044852:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099786.044863:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099786.044879:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d602f5b x1851885417231680/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099786.044906:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099786.044909:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099786.045059:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417231552:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099786.045143:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099786.045149:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417231616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099786.045260:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231680:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099786.053828:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417231616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099786.053848:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417231552:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099786.056908:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099786.056919:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099786.056928:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231680:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099786.300576:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099786.814373:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099787.328039:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099787.836479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099788.350295:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099788.860463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099789.372672:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099789.884479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099790.396761:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099790.911573:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099791.164537:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099791.164545:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099791.164548:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099791.164560:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4295 00000100:00080000:0.0:1766099791.164563:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099791.164590:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099791.164602:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099791.164621:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002326d21d x1851885417231872/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099791.164646:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099791.164649:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099791.164768:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417231744:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099791.164872:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099791.164879:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417231808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099791.165057:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231872:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099791.170813:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417231808:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099791.170832:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417231744:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099791.170991:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099791.171011:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099791.171017:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417231872:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099791.420631:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099791.932492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099792.444583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099792.956500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099793.469837:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099793.983525:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099794.493751:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099795.004492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099795.516626:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099796.028564:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099796.284596:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099796.284603:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099796.284606:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099796.284616:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4300 00000100:00080000:1.0:1766099796.284619:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099796.284625:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099796.284634:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099796.284647:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417232064/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099796.284671:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099796.284674:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099796.284700:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417232000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099796.284798:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766099796.284806:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417231936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099796.284910:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099796.291860:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417231936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099796.291902:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417232000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099796.299968:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099796.299981:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099796.299990:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232064:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099796.540663:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099797.052520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099797.567634:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099798.077421:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099798.598747:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099799.100478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099799.612624:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099800.124493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099800.636582:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099801.148922:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766099801.405531:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417232128:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099801.406338:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099801.406347:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417232192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099801.411034:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099801.411044:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099801.411048:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099801.411059:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4305 00000100:00080000:2.0:1766099801.411062:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766099801.411068:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099801.411079:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099801.411111:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007b4dad67 x1851885417232256/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099801.411138:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099801.411141:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099801.411228:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232256:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099801.412756:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417232192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099801.412795:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417232128:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099801.419526:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099801.419542:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099801.419555:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232256:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099801.660583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099802.172658:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099802.689446:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099803.196519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099803.711577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099804.226392:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099804.734554:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099805.244520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099805.756652:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099806.268515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099806.526315:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099806.526339:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099806.526343:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099806.526398:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4310 00000100:00080000:0.0:1766099806.526402:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099806.526410:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099806.526423:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099806.526443:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d47f608b x1851885417232448/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099806.526462:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099806.526466:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099806.526566:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232320:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099806.526678:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099806.526683:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417232384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099806.527548:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232448:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099806.533928:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232320:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099806.533971:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417232384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099806.535555:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099806.535565:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099806.535574:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232448:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766099806.782477:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099807.292478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099807.807534:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099808.316642:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099808.828571:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099809.340472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099809.853881:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099810.364527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099810.877297:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099811.389881:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099811.646996:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099811.647004:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099811.647007:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099811.647017:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4315 00000100:00080000:1.0:1766099811.647020:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766099811.647028:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099811.647028:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099811.647042:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099811.647059:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f763163e x1851885417232640/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099811.647081:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099811.647084:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099811.647107:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417232576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099811.647252:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232640:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099811.651223:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417232576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099811.652188:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099811.652542:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099811.652547:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099811.652551:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232640:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099811.900616:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099812.414048:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099812.924875:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099813.438635:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099813.948868:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099814.460500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099814.972590:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099815.484480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099815.996577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099816.508883:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099816.764894:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099816.764901:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099816.764904:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099816.764915:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4320 00000100:00080000:0.0:1766099816.764918:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099816.764924:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099816.764933:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099816.764948:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417232832/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099816.764970:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099816.764973:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099816.765137:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232832:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099816.765320:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099816.765383:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766099816.765390:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417232768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099816.769002:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417232768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099816.769020:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232704:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099816.769881:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099816.769901:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099816.769907:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417232832:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099817.021108:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099817.532488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099818.044613:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099818.556489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099819.068819:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099819.580495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099820.093136:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099820.604476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099821.116761:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099821.628702:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099821.885008:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099821.885014:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099821.885018:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099821.885028:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4325 00000100:00080000:0.0:1766099821.885032:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099821.885038:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099821.885048:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099821.885063:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417233024/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099821.885084:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099821.885087:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099821.885108:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232896:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099821.885196:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099821.885202:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417232960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099821.885334:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233024:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099821.898510:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417232960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099821.898838:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099821.898844:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099821.898849:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233024:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099821.899665:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417232896:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766099822.141253:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099822.652979:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099823.164916:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099823.676736:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099824.189734:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099824.704253:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099825.212973:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099825.724479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099826.236941:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099826.748473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099827.004531:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099827.004539:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099827.004542:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099827.004552:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4330 00000100:00080000:0.0:1766099827.004556:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766099827.004561:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099827.004570:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099827.004586:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2c84c91 x1851885417233216/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099827.004652:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233216:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766099827.004750:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099827.004754:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099827.004778:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099827.004827:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099827.004833:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417233088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099827.007328:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417233088:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099827.007350:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099827.007747:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099827.007753:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099827.007757:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233216:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099827.260727:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099827.772469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099828.285101:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099828.796509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099829.308581:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099829.820472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099830.333100:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099830.844489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099831.360213:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099831.868486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099832.124663:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099832.124670:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099832.124674:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099832.124684:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4336 00000100:00080000:0.0:1766099832.124687:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1766099832.124688:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417233280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099832.124692:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099832.124702:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099832.124715:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000024975221 x1851885417233408/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099832.124739:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099832.124741:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099832.124769:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099832.125020:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233408:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099832.134924:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417233280:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099832.136902:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233344:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099832.137107:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099832.137112:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099832.137117:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233408:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099832.380737:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099832.893376:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099833.406909:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099833.916481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099834.432752:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099834.943343:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099835.453121:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099835.964771:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099836.477524:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099836.988494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099837.244577:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099837.244584:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099837.244588:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099837.244598:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4341 00000100:00080000:3.0:1766099837.244601:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766099837.244606:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099837.244616:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099837.244628:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004415ca00 x1851885417233600/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099837.244649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099837.244652:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099837.244709:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233600:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099837.244850:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417233472:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099837.244949:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099837.244957:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417233536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099837.259564:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099837.259575:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099837.259584:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233600:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099837.259851:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417233536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099837.259873:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417233472:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766099837.502168:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099838.014701:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099838.527807:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099839.036497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099839.554342:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099840.060508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099840.572627:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099841.084580:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099841.599809:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099842.108576:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766099842.367639:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417233664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099842.369013:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099842.369021:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099842.369025:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099842.369033:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4346 00000100:00080000:1.0:1766099842.369036:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099842.369042:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099842.369053:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099842.369087:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417233792/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099842.369112:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099842.369115:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099842.369179:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233792:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099842.369292:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099842.378693:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417233664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099842.382169:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233728:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099842.387872:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099842.387884:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099842.387894:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233792:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099842.620606:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099843.132582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766099843.645203:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099844.156429:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099844.676920:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099845.180484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099845.693570:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099846.227298:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099846.716562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099847.230305:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099847.484936:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099847.484944:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099847.484948:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099847.484958:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4351 00000100:00080000:0.0:1766099847.484962:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 6s 00000100:00080000:0.0:1766099847.484968:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099847.484979:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099847.484995:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417233984/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099847.485018:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099847.485051:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099847.485094:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099847.485187:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099847.485193:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417233856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099847.485482:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233984:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099847.489655:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417233920:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099847.491810:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417233856:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099847.499985:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099847.499997:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099847.500007:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417233984:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099847.740554:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099848.252487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099848.765542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099849.277159:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099849.789477:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099850.302197:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099850.813052:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099851.324486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099851.836588:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099852.348481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099852.604868:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099852.604891:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099852.604895:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099852.604905:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4356 00000100:00080000:1.0:1766099852.604910:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:1.0:1766099852.604915:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099852.604927:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099852.604943:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd44c x1851885417234176/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099852.604966:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099852.604970:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099852.605100:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417234048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099852.605188:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099852.605193:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417234112:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099852.605443:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234176:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099852.639069:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417234112:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099852.639085:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417234048:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099852.639348:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099852.639373:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099852.639377:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234176:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099852.860617:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099853.372583:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099853.885101:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099854.396489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099854.909146:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099855.420463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099855.932603:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099856.444457:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099856.957865:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099857.468472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099857.724537:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099857.724545:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1766099857.724550:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417234240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099857.724550:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099857.724560:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4361 00000100:00080000:1.0:1766099857.724565:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1766099857.724571:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099857.724583:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099857.724603:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000baf3fdb7 x1851885417234368/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099857.724630:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099857.724633:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099857.724651:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099857.724658:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417234304:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099857.724881:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234368:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099857.726968:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417234240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099857.727299:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417234304:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099857.727417:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099857.727423:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099857.727428:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234368:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099857.980657:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099858.492490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099859.006897:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099859.518578:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099860.030386:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099860.545323:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099861.054780:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099861.565273:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099862.077740:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099862.588832:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099862.845943:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099862.845951:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099862.845954:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099862.845965:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4366 00000100:00080000:0.0:1766099862.845983:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099862.845989:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099862.845998:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099862.846015:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ff0630b x1851885417234560/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099862.846044:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099862.846047:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099862.846075:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417234496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099862.846174:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099862.846180:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417234432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099862.846347:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234560:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099862.852489:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417234496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:2.0:1766099862.864701:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00100000:0.0:1766099862.873163:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417234432:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099862.897087:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099862.897100:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099862.897111:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234560:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099863.100871:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099863.612447:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099864.125292:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099864.638104:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099865.148554:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099865.660499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099866.172563:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099866.690175:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099867.196532:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099867.708942:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099867.964788:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099867.964795:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099867.964798:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099867.964808:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4371 00000100:00080000:0.0:1766099867.964811:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099867.964817:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099867.964828:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099867.964844:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417234752/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099867.964870:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099867.964915:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099867.964942:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417234688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099867.965034:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099867.965041:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417234624:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099867.965137:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234752:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099867.969922:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417234688:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099867.971399:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099867.971408:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099867.971416:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234752:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099867.971499:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417234624:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099868.220581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099868.735006:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099869.244724:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099869.756594:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099870.271472:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099870.780462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099871.292577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099871.804493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099872.321947:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099872.828563:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099873.088159:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099873.088168:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099873.088172:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099873.088182:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4377 00000100:00080000:3.0:1766099873.088185:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099873.088192:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099873.088217:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099873.088236:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dfb67b84 x1851885417234944/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099873.088262:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099873.088266:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099873.088287:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417234880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099873.088413:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099873.088420:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417234816:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099873.088680:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234944:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099873.099577:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417234880:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099873.103774:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099873.103785:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099873.103794:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417234944:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099873.103894:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417234816:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099873.340583:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099873.854865:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099874.364547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099874.876482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099875.391991:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099875.905754:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099876.412567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099876.925405:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099877.438061:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099877.948464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099878.206275:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099878.206282:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099878.206285:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099878.206295:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4382 00000100:00080000:1.0:1766099878.206298:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099878.206303:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099878.206313:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099878.206327:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417235136/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099878.206350:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099878.206377:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099878.206447:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417235008:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099878.206529:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099878.206534:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417235072:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099878.206613:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235136:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099878.214964:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417235072:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099878.214982:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417235008:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766099878.215071:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099878.215076:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099878.215080:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235136:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099878.460579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099878.975074:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099879.487060:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099879.996599:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099880.509830:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099881.020473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099881.532927:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099882.044482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099882.557147:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099883.068494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099883.324657:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099883.324665:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099883.324668:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099883.324677:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4387 00000100:00080000:3.0:1766099883.324680:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099883.324685:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099883.324695:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099883.324710:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417235328/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099883.324787:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099883.324790:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099883.324901:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235328:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099883.325741:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417235200:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099883.325800:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099883.325807:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417235264:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766099883.328188:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099883.328200:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099883.328210:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235328:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099883.328950:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417235264:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099883.331939:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417235200:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099883.581679:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099884.092473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099884.604788:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099885.116649:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099885.628524:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099886.140438:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099886.652569:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099887.164473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099887.679483:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099888.188924:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099888.448940:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099888.448946:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099888.448950:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099888.448959:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4392 00000100:00080000:2.0:1766099888.448962:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099888.448967:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099888.448976:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099888.448988:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417235520/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099888.449011:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099888.449014:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099888.449037:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417235456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099888.449121:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099888.449126:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417235392:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099888.449349:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235520:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099888.453720:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417235456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099888.456615:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417235392:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099888.466833:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099888.466845:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099888.466855:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235520:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099888.703145:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099889.212572:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099889.724994:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099890.236564:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099890.748602:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099891.260488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099891.773447:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099892.284620:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099892.798789:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099893.309812:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099893.567939:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099893.567946:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099893.567949:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099893.567959:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4397 00000100:00080000:0.0:1766099893.567962:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099893.567967:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099893.567978:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099893.568008:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005ed5ddd4 x1851885417235712/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099893.568033:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099893.568036:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099893.568058:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417235648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099893.568137:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099893.568143:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417235584:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099893.568261:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235712:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099893.577643:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417235584:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099893.577667:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417235648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099893.577801:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099893.577806:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099893.577810:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235712:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099893.820620:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099894.332462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099894.844905:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099895.357397:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099895.868528:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099896.380495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099896.892690:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099897.404914:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099897.916746:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099898.428492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099898.684876:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099898.684883:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099898.684886:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099898.684896:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4402 00000100:00080000:2.0:1766099898.684899:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099898.684906:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099898.684916:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099898.684931:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417235904/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099898.684959:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099898.684962:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099898.684988:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417235840:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099898.685080:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766099898.685086:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417235776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099898.685873:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235904:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766099898.695869:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099898.695879:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099898.695889:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417235904:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766099898.696044:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417235776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099898.696058:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417235840:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099898.941310:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099899.452474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099899.964666:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099900.477774:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099900.988580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099901.500560:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099902.012854:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766099902.524909:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099903.037465:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099903.552231:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099903.805744:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099903.805752:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099903.805755:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099903.805765:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4407 00000100:00080000:3.0:1766099903.805769:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099903.805775:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099903.805785:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099903.805800:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417236096/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099903.805847:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099903.805851:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099903.805929:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417235968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099903.806024:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099903.806031:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417236032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099903.807203:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236096:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099903.816041:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417235968:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099903.817495:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417236032:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766099903.822044:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766099903.822075:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766099903.822085:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236096:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099904.060592:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099904.572633:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099905.085178:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099905.598782:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099906.108503:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099906.620473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099907.132553:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099907.644467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099908.156587:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099908.668593:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099908.924703:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099908.924712:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099908.924716:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099908.924729:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4412 00000100:00080000:1.0:1766099908.924733:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099908.924740:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099908.924754:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099908.924773:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417236288/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099908.924796:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099908.924799:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099908.924897:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099908.925104:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417236160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099908.925143:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099908.925149:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417236224:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099908.930862:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099908.930878:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099908.930890:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236288:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099908.930968:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417236160:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099908.932630:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417236224:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099909.181182:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099909.692886:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099910.204749:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099910.716529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099911.231687:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099911.740543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099912.252583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099912.764554:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099913.276584:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099913.788498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099914.045807:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099914.045815:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099914.045818:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099914.045828:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4418 00000100:00080000:1.0:1766099914.045831:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099914.045837:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766099914.045842:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417236416:192.168.203.152@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1766099914.045848:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099914.045866:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417236480/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099914.045894:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099914.045899:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099914.045934:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417236352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099914.046116:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236480:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1766099914.051614:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099914.051624:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099914.051632:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236480:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099914.053621:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417236352:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099914.063522:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417236416:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099914.300529:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099914.812648:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099915.324582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099915.836481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099916.348494:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099916.860482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099917.373482:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099917.885626:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099918.398787:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099918.909335:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099919.168012:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099919.168032:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099919.168036:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099919.168047:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4423 00000100:00080000:0.0:1766099919.168050:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099919.168056:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099919.168066:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099919.168084:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005ed5ddd4 x1851885417236672/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099919.168109:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099919.168113:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099919.168138:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417236608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099919.168229:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766099919.168235:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417236544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099919.168342:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236672:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099919.173873:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417236544:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099919.173893:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1766099919.173899:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417236608:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099919.173899:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099919.173907:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005ed5ddd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236672:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099919.420557:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099919.932980:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099920.444579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099920.956529:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099921.468582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099921.980491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099922.492692:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099923.005065:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099923.516625:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099924.028717:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099924.284748:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099924.284757:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099924.284760:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099924.284773:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4428 00000100:00080000:1.0:1766099924.284777:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099924.284783:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099924.284794:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099924.284810:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417236864/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099924.284831:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099924.284835:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099924.284897:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099924.286011:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417236800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099924.287566:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417236736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099924.294812:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099924.294823:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099924.294831:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417236864:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766099924.297163:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417236736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099924.299745:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417236800:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099924.540624:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099925.052500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099925.564594:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099926.076514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099926.590525:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099927.100520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099927.617456:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099928.124516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099928.636620:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099929.150682:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099929.404856:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099929.404863:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099929.404866:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099929.404876:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4433 00000100:00080000:2.0:1766099929.404880:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099929.404885:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099929.404895:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099929.404910:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ba671fa0 x1851885417237056/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099929.404931:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099929.404935:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766099929.404955:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417236928:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099929.405032:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766099929.405039:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417236992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766099929.405213:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237056:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099929.415008:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417236992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099929.415046:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417236928:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099929.417202:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099929.417213:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099929.417221:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237056:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766099929.665680:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099930.172533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099930.685430:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099931.196580:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099931.708982:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099932.220601:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099932.732577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099933.244519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099933.756632:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099934.268505:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099934.525939:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099934.525962:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099934.525965:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099934.525982:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4438 00000100:00080000:3.0:1766099934.525986:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099934.525996:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099934.526007:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099934.526022:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417237248/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099934.526044:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099934.526047:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099934.526073:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417237184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099934.526106:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417237120:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099934.526341:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237248:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766099934.529465:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1766099934.529466:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417237184:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099934.529473:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099934.529476:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237248:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766099934.529563:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417237120:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099934.781058:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099935.292571:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099935.804581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099936.316565:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099936.829154:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099937.340417:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099937.852575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099938.364495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099938.876570:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099939.388477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766099939.647625:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099939.647634:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766099939.647655:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766099939.647668:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4443 00000100:00080000:2.0:1766099939.647672:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1766099939.647679:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766099939.647691:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766099939.647707:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417237440/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766099939.647732:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766099939.647735:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099939.647815:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237440:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766099939.647961:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417237312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099939.647994:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766099939.648000:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417237376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099939.651216:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417237376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766099939.651253:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417237312:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099939.651407:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099939.651412:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099939.651417:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237440:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099939.901677:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099940.412586:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099940.927841:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099941.436498:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099941.951222:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099942.460482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099942.972602:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099943.485548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099943.996567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099944.509373:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099944.764529:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099944.764537:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099944.764540:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766099944.764550:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417237504:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099944.764550:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4448 00000100:00080000:3.0:1766099944.764553:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099944.764557:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099944.764566:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099944.764580:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006a1d607 x1851885417237632/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099944.764604:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099944.764607:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099944.764625:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417237568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099944.764718:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766099944.767611:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099944.767635:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099944.767643:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237632:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099944.769742:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417237568:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099944.769942:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417237504:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099945.020673:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099945.532492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099946.044546:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099946.556513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099947.069216:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099947.580633:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099948.093487:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099948.605185:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099949.117638:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099949.629055:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766099949.884556:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099949.884566:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766099949.884569:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766099949.884581:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4453 00000100:00080000:3.0:1766099949.884602:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1766099949.884609:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766099949.884622:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766099949.884641:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417237824/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766099949.884670:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766099949.884673:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099949.884702:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417237760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099949.884774:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237824:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766099949.884795:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766099949.884804:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417237696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099949.889750:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417237696:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099949.889773:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417237760:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099949.889813:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766099949.889819:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766099949.889824:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417237824:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099950.140615:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099950.652528:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099951.164591:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099951.676496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099952.188573:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1766099952.700495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099953.212560:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099953.724500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099954.236573:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099954.748557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766099955.006168:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417237888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099955.008417:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099955.008460:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417237952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099955.009603:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099955.009612:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099955.009616:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099955.009624:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4458 00000100:00080000:1.0:1766099955.009627:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099955.009633:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099955.009643:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099955.009671:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000174f4196 x1851885417238016/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099955.009700:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099955.009712:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766099955.009786:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238016:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099955.014925:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417237888:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099955.015774:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417237952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099955.016201:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099955.016206:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099955.016211:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238016:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766099955.260598:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099955.772549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099956.293342:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099956.796471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099957.308569:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099957.820486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099958.332811:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099958.844591:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099959.356706:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099959.868610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099960.124591:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099960.124598:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099960.124602:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099960.124625:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4464 00000100:00080000:0.0:1766099960.124628:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099960.124634:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099960.124643:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099960.124659:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059092741 x1851885417238208/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099960.124689:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099960.124692:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099960.124717:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099960.124722:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099960.124776:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238080:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099960.135559:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099960.135571:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099960.135580:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059092741 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238208:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099960.144271:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238144:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099960.145029:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000030e2f50e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238080:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099960.381180:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099960.892860:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099961.405031:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099961.916473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099962.428583:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099962.944842:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099963.455335:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099963.964486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099964.483585:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099964.988476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766099965.244537:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099965.244546:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766099965.244550:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766099965.244560:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4469 00000100:00080000:0.0:1766099965.244564:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766099965.244582:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766099965.244592:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766099965.244607:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088a8ce3f x1851885417238400/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766099965.244634:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766099965.244638:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099965.244660:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099965.244679:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099965.244757:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238400:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099965.251243:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000512a78fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238272:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099965.253880:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238336:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099965.255117:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099965.255127:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099965.255135:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088a8ce3f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238400:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099965.502260:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099966.012494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099966.525764:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099967.036969:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099967.549722:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099968.060480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099968.573708:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099969.084548:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099969.597183:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766099970.108530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099970.364590:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099970.364599:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099970.364604:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099970.364618:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4474 00000100:00080000:1.0:1766099970.364622:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099970.364629:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099970.364642:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099970.364660:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417238592/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766099970.364752:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238592:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099970.365540:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099970.365599:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099970.365605:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417238528:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099970.365692:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099970.365696:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766099970.370264:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417238528:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766099970.370540:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099970.370546:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099970.370550:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238592:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099970.371791:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238464:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766099970.620676:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099971.132556:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099971.646049:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099972.160275:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099972.668821:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099973.180472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099973.692530:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099974.204475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099974.716538:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099975.228473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099975.484543:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099975.484551:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099975.484554:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099975.484566:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4479 00000100:00080000:1.0:1766099975.484571:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1766099975.484579:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099975.484579:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099975.484593:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099975.484610:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417238784/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099975.484652:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099975.484655:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099975.484675:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766099975.491616:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238784:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099975.493824:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238720:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099975.494230:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238656:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099975.498302:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099975.498313:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099975.498322:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238784:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099975.741991:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099976.253224:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099976.768516:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099977.276469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099977.788601:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099978.300540:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099978.812778:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099979.324481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099979.837753:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099980.351380:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099980.604939:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099980.604948:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099980.604951:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099980.604962:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4484 00000100:00080000:1.0:1766099980.604965:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099980.604972:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099980.604982:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099980.604997:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c23332fe x1851885417238976/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766099980.605101:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238976:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099980.605269:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099980.605530:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099980.605535:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099980.605556:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238912:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099980.612257:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099980.612268:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099980.612277:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417238976:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099980.613492:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417238848:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099980.615930:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417238912:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766099980.860598:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099981.372565:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099981.886708:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099982.396524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099982.908550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099983.420544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099983.935151:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099984.444946:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099984.956573:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099985.468492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099985.724599:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099985.724607:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099985.724611:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099985.724622:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4489 00000100:00080000:1.0:1766099985.724626:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099985.724631:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000100:00100000:0.0:1766099985.724638:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239040:192.168.203.152@tcp:400:kworker/u8:1.0 00000008:00100000:1.0:1766099985.724640:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099985.724658:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c9026f0 x1851885417239168/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099985.724688:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099985.724692:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099985.724712:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417239104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099985.724864:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239168:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766099985.730886:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239040:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766099985.731002:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417239104:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766099985.738935:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099985.738944:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099985.738953:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239168:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766099985.980585:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099986.492473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099987.004593:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099987.516567:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099988.030417:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099988.540641:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099989.052789:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099989.564483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099990.076767:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099990.588481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099990.844513:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099990.844520:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099990.844523:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766099990.844533:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4494 00000100:00080000:1.0:1766099990.844536:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766099990.844554:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099990.844563:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099990.844579:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417239360/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766099990.844592:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239232:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766099990.844657:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099990.844727:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766099990.844739:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417239296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766099990.846654:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099990.846663:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1766099990.854590:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766099990.854615:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766099990.854626:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239360:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766099990.856160:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417239296:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766099990.856174:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239232:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766099991.102681:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099991.612908:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099992.124566:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099992.636564:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099993.149492:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099993.660773:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099994.175476:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099994.686430:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099995.198426:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099995.708793:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766099995.967403:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099995.967412:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766099995.967416:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766099995.967417:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766099995.967424:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4499 00000100:00080000:1.0:1766099995.967426:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766099995.967434:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766099995.967447:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766099995.967464:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417239552/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766099995.967477:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766099995.967480:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766099995.967504:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417239488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766099995.967950:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239552:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766099995.982325:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417239488:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766099995.982987:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239424:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766099995.983595:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766099995.983602:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766099995.983609:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239552:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766099996.220974:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099996.733733:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099997.245618:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099997.758438:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766099998.271441:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099998.781899:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766099999.296308:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766099999.804522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100000.318994:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100000.828639:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766100001.084873:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100001.085198:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100001.085205:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417239680:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766100001.085461:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100001.085467:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100001.085470:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100001.085475:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4505 00000100:00080000:1.0:1766100001.085478:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100001.085483:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100001.085494:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100001.085512:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f7eb885 x1851885417239744/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100001.085535:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100001.085538:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100001.085630:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239744:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100001.096126:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417239680:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100001.096146:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417239616:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100001.097887:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100001.097898:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100001.097916:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239744:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766100001.340684:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100001.852527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100002.369677:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100002.877553:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100003.389390:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100003.903927:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100004.413505:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100004.925403:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100005.438391:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100005.948456:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100006.204800:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100006.204808:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100006.204811:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100006.204821:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4510 00000100:00080000:3.0:1766100006.204824:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1766100006.204829:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417239808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100006.204829:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100006.204839:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100006.204852:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417239936/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100006.204925:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239936:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100006.204925:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100006.204930:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417239872:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100006.206447:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100006.206454:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766100006.212936:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100006.212950:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100006.212961:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417239936:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100006.214897:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417239808:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100006.216157:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417239872:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766100006.460667:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100006.978004:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100007.488651:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100007.996520:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100008.508710:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100009.020499:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100009.533176:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100010.045844:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100010.557458:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100011.068530:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100011.324595:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100011.324602:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100011.324606:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100011.324619:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4515 00000100:00080000:1.0:1766100011.324622:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100011.324627:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100011.324651:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100011.324668:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000094c69f2b x1851885417240128/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100011.324696:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100011.324700:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100011.324792:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417240064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100011.324934:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766100011.324941:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417240000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100011.328023:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240128:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100011.329434:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417240000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100011.329457:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417240064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100011.332729:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100011.332738:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100011.332748:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240128:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100011.580544:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100012.092708:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100012.604664:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100013.116480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100013.628579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100014.144736:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100014.654635:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100015.166945:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100015.676548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100016.188490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100016.444587:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100016.444594:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100016.444597:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100016.444606:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4520 00000100:00080000:2.0:1766100016.444609:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100016.444614:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100016.444623:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100016.444636:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c471d159 x1851885417240320/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100016.444657:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100016.444660:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100016.444683:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417240256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100016.444757:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100016.444763:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417240192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100016.444881:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240320:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100016.452249:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417240256:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100016.452276:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417240192:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1766100016.452949:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:0.0:1766100016.452957:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100016.452960:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100016.452963:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240320:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766100016.704022:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100017.213217:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100017.724605:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100018.236549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100018.748543:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100019.260477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100019.774441:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100020.284898:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100020.796587:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100021.309251:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766100021.566895:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417240384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766100021.567407:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100021.567415:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417240448:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100021.568196:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100021.568204:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100021.568209:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100021.568215:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4525 00000100:00080000:2.0:1766100021.568219:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100021.568225:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100021.568239:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100021.568263:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000db8badc0 x1851885417240512/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100021.568331:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240512:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766100021.568859:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100021.568864:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100021.576417:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417240448:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100021.576437:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417240384:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766100021.576656:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100021.576662:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100021.576666:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240512:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766100021.821687:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100022.332985:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100022.848103:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100023.356605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100023.869508:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100024.382819:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100024.892549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100025.404522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100025.916872:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100026.429944:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100026.684563:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100026.684572:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100026.684575:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100026.684587:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4530 00000100:00100000:3.0:1766100026.684588:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417240576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100026.684591:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100026.684610:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100026.684620:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100026.684639:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001dc2eb3d x1851885417240704/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100026.684666:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100026.684670:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100026.684680:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100026.684690:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417240640:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100026.684777:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240704:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100026.691837:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417240576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100026.692542:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417240640:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100026.692583:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100026.692588:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100026.692593:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240704:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100026.940540:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100027.452557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100027.964562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100028.483891:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100028.990311:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100029.501815:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100030.012562:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100030.524563:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100031.040518:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100031.548462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100031.804575:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100031.804582:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100031.804586:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100031.804595:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4535 00000100:00080000:1.0:1766100031.804598:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100031.804603:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100031.804613:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100031.804629:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417240896/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100031.804652:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100031.804656:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100031.804702:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766100031.804705:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417240832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100031.804796:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100031.804805:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417240768:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100031.807424:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100031.807434:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100031.807442:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417240896:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766100031.808385:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417240832:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100031.810848:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417240768:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766100032.061028:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100032.576803:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100033.086433:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100033.596478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100034.109631:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100034.620640:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100035.136115:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100035.646842:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100036.156661:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100036.668536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100036.929544:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100036.929552:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100036.929557:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100036.929570:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4540 00000100:00080000:2.0:1766100036.929574:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100036.929581:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100036.929593:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100036.929608:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417241088/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100036.929653:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100036.929657:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100036.929831:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100036.929932:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100036.929937:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417240960:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100036.929964:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241088:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100036.934172:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241024:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766100036.941093:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100036.941104:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100036.941113:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241088:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100036.942458:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417240960:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766100037.192729:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100037.692562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100038.209222:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100038.716482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100039.228732:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100039.743746:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100040.252737:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100040.764467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100041.276802:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100041.789734:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100042.047837:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100042.047846:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100042.047849:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100042.047860:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4546 00000100:00080000:2.0:1766100042.047863:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100042.047868:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100042.047878:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100042.047895:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417241280/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100042.047918:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100042.047922:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100042.047948:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100042.048034:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100042.048040:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417241152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100042.048251:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241280:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100042.053203:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417241152:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100042.053228:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241216:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766100042.053523:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100042.053528:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100042.053533:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241280:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766100042.301146:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100042.813940:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100043.324548:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100043.837308:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100044.348542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100044.860476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100045.372803:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100045.884571:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100046.397663:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100046.908545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100047.164996:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100047.165004:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100047.165007:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100047.165019:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4551 00000100:00080000:2.0:1766100047.165022:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100047.165028:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100047.165038:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100047.165053:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e377dbf8 x1851885417241472/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100047.165078:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100047.165082:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100047.165108:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241408:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100047.165190:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100047.165196:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417241344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100047.166387:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241472:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100047.167384:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241408:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100047.168130:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100047.168136:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100047.168142:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241472:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100047.170007:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417241344:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766100047.421183:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100047.933142:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100048.444981:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100048.959550:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100049.470423:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100049.982275:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100050.492664:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100051.004515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100051.518718:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100052.030928:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100052.285345:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100052.285393:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100052.285398:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100052.285414:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4556 00000100:00080000:1.0:1766100052.285419:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100052.285426:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100052.285441:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100052.285461:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417241664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100052.285479:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100052.285483:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100052.285550:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417241600:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100052.285676:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100052.285683:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417241536:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100052.286596:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241664:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766100052.290845:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100052.290860:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100052.290872:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241664:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766100052.291547:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417241536:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100052.291568:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417241600:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766100052.546121:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100053.052562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100053.564550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100054.078016:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100054.589076:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100055.100491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100055.617834:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100056.124462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100056.636588:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100057.149705:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100057.404819:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100057.404827:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100057.404830:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100057.404841:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4561 00000100:00080000:2.0:1766100057.404844:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100057.404850:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100057.404861:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100057.404876:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417241856/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100057.404901:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100057.404905:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100057.404933:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241792:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100057.404986:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241856:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766100057.405264:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417241728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766100057.414968:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417241728:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100057.415403:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100057.415412:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100057.415433:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417241856:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100057.415466:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417241792:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766100057.660678:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100058.172550:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100058.684600:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100059.196811:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100059.708561:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100060.220508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100060.734550:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100061.245683:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100061.756581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100062.268490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100062.525223:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100062.525231:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100062.525235:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100062.525247:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4566 00000100:00080000:1.0:1766100062.525250:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100062.525256:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100062.525267:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100062.525284:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417242048/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100062.525304:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100062.525307:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100062.525340:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417241984:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100062.525490:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100062.525497:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417241920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100062.526068:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242048:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100062.532940:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417241920:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100062.532968:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417241984:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100062.533596:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100062.533604:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100062.533609:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242048:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766100062.780670:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100063.292581:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100063.804576:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100064.316516:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100064.829403:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100065.340924:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100065.853865:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100066.364574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100066.876553:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100067.388455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766100067.644621:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417242112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100067.645489:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100067.645498:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417242176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1766100067.645718:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100067.645723:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100067.645726:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100067.645731:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4571 00000100:00080000:3.0:1766100067.645733:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100067.645739:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100067.645748:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100067.645764:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417242240/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100067.645786:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100067.645789:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100067.645956:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242240:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100067.657713:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417242176:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100067.660929:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417242112:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1766100067.661666:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100067.661672:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100067.661678:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242240:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766100067.902818:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100068.412582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100068.924593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100069.436480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100069.949836:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100070.462134:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100070.974499:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100071.484493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100071.996674:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100072.508474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100072.764581:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100072.764587:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100072.764590:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100072.764600:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4576 00000100:00080000:1.0:1766100072.764603:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100072.764608:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100072.764617:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100072.764631:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417242432/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100072.764654:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100072.764657:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100072.764779:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417242304:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100072.764850:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766100072.764856:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417242368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766100072.764944:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242432:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1766100072.773619:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100072.773628:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100072.773637:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242432:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1766100072.773764:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417242368:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100072.773776:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417242304:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766100073.020574:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100073.532838:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100074.045422:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100074.556467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100075.068580:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100075.580489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100076.093809:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100076.604543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100077.116615:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100077.629487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100077.884754:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100077.884760:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100077.884763:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100077.884773:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4581 00000100:00080000:1.0:1766100077.884776:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100077.884781:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100077.884790:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100077.884802:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417242624/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100077.884824:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100077.884827:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100077.884849:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417242560:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100077.884986:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417242496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766100077.885054:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242624:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:2.0:1766100077.899955:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100077.899965:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100077.899975:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242624:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100077.900179:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417242496:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100077.902154:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417242560:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766100078.140727:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100078.652497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100079.164895:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100079.677063:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100080.188584:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100080.700472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100081.212568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100081.724478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100082.236577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100082.749192:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100083.004649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100083.004658:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100083.004662:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100083.004675:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4586 00000100:00080000:2.0:1766100083.004679:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100083.004686:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100083.004715:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100083.004734:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f8448657 x1851885417242816/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100083.004812:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242816:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766100083.005105:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100083.005108:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100083.005129:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417242752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100083.005162:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100083.005166:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417242688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100083.016966:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417242688:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100083.016990:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417242752:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100083.017088:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100083.017093:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100083.017098:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417242816:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100083.260577:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100083.772574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100084.284712:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100084.796687:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100085.308676:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100085.820467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100086.332602:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100086.844464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100087.359618:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100087.868486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766100088.124878:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417242880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100088.125616:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100088.125625:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417242944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766100088.126060:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100088.126066:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100088.126069:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100088.126075:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4592 00000100:00080000:1.0:1766100088.126078:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100088.126092:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100088.126101:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100088.126119:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000094c69f2b x1851885417243008/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100088.126143:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100088.126146:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100088.126668:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243008:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100088.138506:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417242880:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100088.138586:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417242944:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100088.138644:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100088.138649:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100088.138654:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243008:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1766100088.381621:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100088.892490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100089.406048:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100089.917419:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100090.431454:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100090.940582:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100091.452762:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100091.964496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100092.476746:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100092.988454:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100093.244586:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100093.244594:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100093.244598:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100093.244610:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4597 00000100:00080000:1.0:1766100093.244614:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100093.244621:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100093.244631:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100093.244646:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f7eb885 x1851885417243200/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100093.244676:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100093.244679:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100093.244774:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417243072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100093.244884:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100093.244901:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417243136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100093.245692:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243200:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100093.248666:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417243136:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100093.248687:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417243072:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100093.255563:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100093.255576:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100093.255586:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243200:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1766100093.500532:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100094.012507:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100094.527907:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100095.036790:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100095.548530:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100096.060624:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100096.572568:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100097.084471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100097.596642:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100098.108477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100098.366062:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100098.366071:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100098.366074:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100098.366084:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4602 00000100:00080000:2.0:1766100098.366100:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100098.366105:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100098.366115:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100098.366131:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417243392/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100098.366158:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100098.366161:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100098.366189:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417243328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100098.366298:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100098.366304:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417243264:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100098.366435:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243392:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100098.369401:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417243264:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100098.369421:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417243328:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100098.378034:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100098.378045:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100098.378054:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243392:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100098.620571:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100099.132468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100099.645507:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100100.156504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100100.668991:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100101.180738:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100101.694262:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100102.204470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100102.722463:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100103.228485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100103.486721:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100103.486727:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100103.486745:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100103.486755:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4607 00000100:00080000:1.0:1766100103.486758:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100103.486764:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100103.486775:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100103.486793:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417243584/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100103.486819:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100103.486823:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100103.486883:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100103.487465:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417243456:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100103.487529:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100103.487537:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417243520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100103.493648:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100103.493660:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100103.493671:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243584:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100103.493766:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417243520:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100103.493776:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417243456:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1766100103.741203:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100104.253881:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100104.767438:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100105.276475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100105.788580:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100106.300476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100106.812680:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100107.324567:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100107.836526:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100108.348481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100108.604676:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100108.604683:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100108.604686:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766100108.604691:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417243648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766100108.604691:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4612 00000100:00080000:1.0:1766100108.604695:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100108.604699:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100108.604710:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100108.604722:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9051765 x1851885417243776/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100108.604737:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100108.604740:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100108.604755:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417243712:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100108.604809:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243776:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766100108.609973:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100108.609984:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100108.609994:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243776:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100108.615582:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417243648:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100108.615582:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417243712:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766100108.860573:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100109.372488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100109.886284:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100110.396495:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100110.908561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100111.420559:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100111.933295:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100112.444561:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100112.956715:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100113.468575:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100113.724522:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100113.724529:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100113.724534:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100113.724545:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4617 00000100:00080000:2.0:1766100113.724548:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100113.724566:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100113.724576:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100113.724592:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417243968/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100113.724616:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100113.724619:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100113.724640:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417243840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1766100113.724722:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100113.724728:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417243904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100113.724915:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243968:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766100113.730480:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417243904:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1766100113.730502:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417243840:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766100113.730687:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100113.730692:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100113.730697:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417243968:192.168.203.152@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1766100113.980504:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100114.492544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100115.007919:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100115.516483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100116.028554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100116.540639:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100117.052732:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100117.564480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100118.076745:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100118.588794:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100118.844578:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100118.844586:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100118.844589:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100118.844599:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4622 00000100:00080000:2.0:1766100118.844602:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100118.844608:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100118.844618:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100118.844637:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6bc72e5 x1851885417244160/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100118.844665:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100118.844670:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100118.844696:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417244096:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100118.844796:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100118.844817:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417244032:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100118.846531:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244160:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100118.851674:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417244032:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100118.851701:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417244096:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100118.857252:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100118.857264:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100118.857273:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244160:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100119.102992:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100119.612988:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100120.124561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100120.636672:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100121.148550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100121.660469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100122.174550:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100122.684478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100123.196982:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100123.708638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100123.977465:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100123.977472:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100123.977476:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100123.977486:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4627 00000100:00080000:1.0:1766100123.977503:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100123.977511:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100123.977521:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100123.977536:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417244352/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766100123.977595:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1766100123.977706:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100123.977710:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100123.977733:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417244288:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100123.977774:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100123.977779:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417244224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766100123.991080:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100123.991091:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100123.991101:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244352:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1766100123.992469:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417244224:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1766100123.992489:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417244288:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1766100124.220612:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100124.732422:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100125.244605:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100125.756473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100126.268593:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100126.780471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100127.293097:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100127.804476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100128.317630:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100128.828481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766100129.085598:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417244416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1766100129.087802:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100129.087809:0:15096:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100129.087812:0:15096:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100129.087822:0:15096:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4633 00000100:00080000:2.0:1766100129.087825:0:15096:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100129.087829:0:15096:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100129.087839:0:15096:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100129.087853:0:15096:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417244544/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100129.087873:0:15096:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100129.087876:0:15096:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100129.087892:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100129.087902:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417244480:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1766100129.088707:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244544:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1766100129.091258:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417244416:192.168.203.152@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1766100129.096102:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100129.096114:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100129.096123:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244544:192.168.203.152@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1766100129.096785:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417244480:192.168.203.152@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1766100129.340739:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100129.854833:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100130.366170:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100130.876515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100131.389509:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100131.900879:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100132.418938:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100132.924476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100133.436886:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100133.952099:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100134.204845:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100134.204852:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100134.204855:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100134.204866:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4638 00000100:00080000:3.0:1766100134.204869:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100134.204889:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100134.204899:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100134.204916:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000edfc3316 x1851885417244736/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100134.204969:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100134.204972:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100134.204998:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417244672:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100134.205077:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100134.205096:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417244608:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100134.205195:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244736:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766100134.211502:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100134.211514:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100134.211524:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000edfc3316 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244736:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100134.211804:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbfb1ac7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417244608:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100134.211819:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52cb2e2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417244672:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766100134.460575:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100134.972508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100135.484547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100135.996480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100136.519405:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100137.020546:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100137.532846:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100138.045783:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100138.559687:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100139.068480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100139.324631:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100139.324655:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100139.324659:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100139.324669:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4643 00000100:00080000:2.0:1766100139.324672:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100139.324678:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100139.324689:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100139.324705:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a51d758 x1851885417244928/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100139.324735:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100139.324739:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100139.324767:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417244864:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100139.324852:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100139.324858:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417244800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100139.325454:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244928:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766100139.328247:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100139.328257:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100139.328265:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417244928:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100139.328427:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417244800:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100139.330648:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417244864:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766100139.583593:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100140.092575:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100140.604661:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100141.116489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100141.630968:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100142.140482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100142.653774:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100143.169078:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100143.676549:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100144.188484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766100144.445215:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417244992:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100144.445927:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100144.445934:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417245056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100144.446717:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100144.446724:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100144.446727:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100144.446735:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4648 00000100:00080000:2.0:1766100144.446737:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100144.446743:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100144.446752:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100144.446771:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000047def236 x1851885417245120/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766100144.446835:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245120:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766100144.447226:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100144.447231:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1766100144.452923:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100144.452935:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100144.452945:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245120:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100144.453882:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417245056:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100144.453898:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417244992:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100144.701428:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100145.212500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100145.727702:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100146.236473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100146.750016:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100147.265154:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100147.778109:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100148.284508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100148.796590:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100149.308487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100149.564684:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100149.564691:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100149.564695:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100149.564704:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4653 00000100:00080000:3.0:1766100149.564707:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100149.564711:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100149.564720:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100149.564733:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417245312/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100149.564754:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100149.564757:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100149.564780:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417245248:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100149.564869:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100149.564888:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417245184:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100149.565076:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245312:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100149.566874:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001161032f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417245248:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100149.585285:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100149.585296:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100149.585305:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245312:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100149.585392:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417245184:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766100149.821096:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100150.332955:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100150.847010:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100151.358962:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100151.868814:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100152.380450:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100152.892558:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100153.404475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100153.916597:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100154.428474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766100154.685060:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417245376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100154.686885:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100154.686891:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100154.686894:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100154.686908:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4658 00000100:00080000:2.0:1766100154.686911:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100154.686917:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100154.686928:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100154.686941:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417245440:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100154.686946:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417245504/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100154.686970:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100154.686973:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100154.687017:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245504:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100154.690508:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417245376:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100154.690744:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417245440:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100154.696997:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100154.697007:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100154.697017:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245504:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766100154.940795:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100155.452502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100155.965282:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100156.477873:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100156.989715:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100157.502222:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100158.013734:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100158.525922:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100159.036549:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100159.548424:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100159.805513:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100159.805520:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100159.805523:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100159.805536:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4663 00000100:00080000:1.0:1766100159.805553:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100159.805560:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100159.805571:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100159.805587:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002b76164b x1851885417245696/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766100159.805650:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245696:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100159.807245:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417245568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100159.807406:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100159.807416:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417245632:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100159.807574:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100159.807579:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100159.819410:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417245632:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100159.819430:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417245568:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766100159.819529:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100159.819534:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100159.819538:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245696:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100160.060874:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100160.572483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100161.084579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100161.596580:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100162.108580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100162.620660:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100163.133553:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100163.644576:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100164.156577:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100164.670252:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100164.924646:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100164.924653:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100164.924656:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100164.924666:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4668 00000100:00080000:0.0:1766100164.924670:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100164.924675:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100164.924684:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100164.924699:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000042205631 x1851885417245888/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100164.924731:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100164.924734:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100164.924757:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417245824:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100164.924911:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245888:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100164.924966:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417245760:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766100164.931967:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100164.931978:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100164.931988:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000042205631 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417245888:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100164.935451:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417245824:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100164.937095:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ff0630b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417245760:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1766100165.181940:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100165.695008:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100166.209293:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100166.716483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100167.229134:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100167.740660:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100168.253581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100168.764569:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100169.277380:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100169.789015:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100170.044609:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100170.044617:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100170.044621:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100170.044631:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4674 00000100:00080000:2.0:1766100170.044634:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100170.044640:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100170.044650:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100170.044666:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000db8badc0 x1851885417246080/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100170.044700:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100170.044703:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100170.044742:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246080:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100170.044961:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417245952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100170.045004:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100170.053518:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417245952:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100170.053593:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246016:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:1.0:1766100170.054176:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:1.0:1766100170.054184:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100170.054187:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100170.054191:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246080:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766100170.300986:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100170.813252:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100171.324567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100171.836574:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100172.348916:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100172.860613:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100173.372798:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100173.884493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100174.401005:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100174.908504:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100175.164550:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100175.164559:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100175.164582:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100175.164595:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4679 00000100:00100000:3.0:1766100175.164596:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417246144:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100175.164599:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100175.164607:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100175.164620:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100175.164669:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b83cf9b1 x1851885417246272/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100175.164693:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100175.164697:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100175.164704:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100175.164713:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417246208:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100175.164916:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246272:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100175.170141:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417246144:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100175.189075:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100175.189088:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100175.189098:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246272:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100175.189201:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417246208:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100175.421246:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100175.934760:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100176.444550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100176.956469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100177.471508:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100177.980440:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100178.492547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100179.004579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100179.516671:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100180.029080:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100180.285266:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100180.285273:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100180.285277:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100180.285286:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4684 00000100:00080000:2.0:1766100180.285289:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100180.285295:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100180.285304:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100180.285319:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009119b13e x1851885417246464/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100180.285342:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100180.285345:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100180.285383:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100180.285403:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246464:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100180.290613:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100180.290636:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417246336:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100180.296425:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100180.296437:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100180.296446:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246464:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100180.296512:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417246336:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100180.298685:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246400:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766100180.541411:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100181.052537:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100181.566145:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100182.076527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100182.589440:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100183.100485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100183.612647:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100184.124510:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100184.637893:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100185.148483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100185.404602:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100185.404611:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100185.404615:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100185.404628:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4689 00000100:00100000:2.0:1766100185.404633:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417246528:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100185.404635:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100185.404646:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100185.404659:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100185.404680:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417246656/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100185.404712:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100185.404715:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100185.404768:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246592:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100185.405160:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246656:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766100185.416421:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100185.416433:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100185.416447:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246656:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100185.416497:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417246528:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100185.416553:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246592:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766100185.660651:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100186.173051:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100186.684643:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100187.198464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100187.709606:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100188.224916:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100188.733680:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100189.244696:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100189.756727:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100190.268598:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100190.524697:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100190.524706:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100190.524710:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100190.524725:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4694 00000100:00080000:3.0:1766100190.524729:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100190.524737:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100190.524750:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100190.524770:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e192245 x1851885417246848/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100190.524804:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100190.524807:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100190.524845:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246784:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100190.524972:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100190.524978:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417246720:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100190.525828:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246848:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766100190.528726:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417246784:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100190.532089:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417246720:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100190.532270:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100190.532275:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100190.532280:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417246848:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100190.781821:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100191.293945:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100191.807754:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100192.316536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100192.829226:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100193.340538:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100193.852646:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100194.364523:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100194.876631:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100195.388606:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100195.645481:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100195.645521:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100195.645525:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100195.645536:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4699 00000100:00080000:2.0:1766100195.645540:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100195.645546:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100195.645558:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100195.645577:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000978f974b x1851885417247040/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766100195.645647:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247040:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100195.646267:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417246912:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100195.646312:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100195.646318:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417246976:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100195.647601:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100195.647609:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1766100195.651794:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100195.651804:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100195.651814:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247040:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100195.651972:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417246976:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100195.651984:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417246912:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766100195.900937:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100196.412901:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100196.926882:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100197.436502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100197.950805:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100198.460484:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100198.972592:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100199.484622:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100199.996564:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100200.508494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100200.764625:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100200.764633:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100200.764636:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100200.764646:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4704 00000100:00080000:2.0:1766100200.764661:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100200.764667:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100200.764676:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100200.764692:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417247232/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100200.764718:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100200.764721:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100200.764765:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247232:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100200.765168:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417247104:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100200.765271:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100200.765277:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417247168:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100200.776822:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100200.776833:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100200.776842:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247232:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100200.777859:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417247104:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100200.779280:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417247168:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100201.021460:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100201.532618:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100202.045099:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100202.558559:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100203.069254:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100203.585567:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100204.101774:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100204.604581:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100205.116581:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100205.628579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100205.884594:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100205.884601:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100205.884604:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100205.884614:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4709 00000100:00080000:0.0:1766100205.884617:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100205.884622:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100205.884631:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100205.884644:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f410160b x1851885417247424/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100205.884667:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100205.884670:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100205.884705:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417247360:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100205.884790:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766100205.884796:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417247296:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100205.884903:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247424:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100205.887387:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417247360:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100205.888903:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417247296:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1766100205.891576:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1766100205.891586:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:2.0:1766100205.891595:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f410160b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247424:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100206.141200:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100206.652464:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100207.165937:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100207.676509:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100208.188599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100208.700585:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100209.212639:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100209.724513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100210.236615:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100210.749481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100211.005483:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100211.005491:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100211.005495:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100211.005507:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4714 00000100:00080000:2.0:1766100211.005512:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1766100211.005513:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417247488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100211.005520:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100211.005534:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100211.005553:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ba671fa0 x1851885417247616/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100211.005582:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100211.005586:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100211.005607:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100211.005615:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417247552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100211.005714:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247616:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100211.010828:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417247552:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100211.010847:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417247488:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100211.010849:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100211.010855:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100211.010858:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ba671fa0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247616:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100211.261311:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100211.772476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100212.284604:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100212.796470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100213.308626:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100213.820622:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100214.335051:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100214.844865:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100215.356979:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100215.868472:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100216.125498:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100216.125506:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100216.125510:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100216.125521:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4720 00000100:00080000:2.0:1766100216.125524:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100216.125530:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100216.125541:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100216.125558:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000428d096e x1851885417247808/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100216.125590:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100216.125594:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100216.125620:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417247680:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100216.125625:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417247744:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100216.126075:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247808:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100216.130314:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417247680:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100216.140329:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100216.140348:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100216.140374:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000428d096e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417247808:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100216.141304:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1d41706 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417247744:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100216.381519:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100216.892483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100217.404585:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100217.916486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100218.428547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100218.940526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100219.454275:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100219.965522:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100220.488838:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100220.988469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100221.249490:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100221.249497:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100221.249501:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100221.249512:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4725 00000100:00080000:3.0:1766100221.249516:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100221.249534:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100221.249545:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100221.249560:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ebab33a4 x1851885417248000/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100221.249586:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100221.249589:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100221.249618:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417247936:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100221.249704:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100221.249710:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417247872:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100221.250456:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248000:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100221.252805:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417247872:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100221.257590:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417247936:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766100221.258460:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100221.258466:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100221.258470:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248000:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100221.500568:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100222.012474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100222.524791:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100223.036613:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100223.548648:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100224.060501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100224.575720:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100225.084807:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100225.596572:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100226.108483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100226.364596:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100226.364605:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100226.364608:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100226.364632:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4730 00000100:00080000:2.0:1766100226.364635:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100226.364641:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100226.364652:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100226.364668:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000744962b0 x1851885417248192/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100226.364732:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248192:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100226.364855:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417248064:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100226.364893:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100226.364899:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417248128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100226.366197:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100226.366205:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100226.372616:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417248128:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100226.378017:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100226.378029:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100226.378037:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000744962b0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248192:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100226.379573:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007b4dad67 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417248064:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766100226.620580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100227.132436:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100227.645970:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100228.156455:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100228.669479:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100229.180503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100229.694542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100230.204705:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100230.718503:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100231.229072:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100231.486327:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100231.486346:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100231.486349:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766100231.486464:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417248256:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100231.486550:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100231.486556:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417248320:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100231.487582:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4735 00000100:00080000:3.0:1766100231.487587:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100231.487592:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100231.487605:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100231.487624:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417248384/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100231.487693:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248384:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766100231.491790:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100231.491798:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100231.505324:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417248320:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100231.505344:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417248256:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100231.505946:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100231.505953:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100231.505958:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248384:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100231.741465:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100232.252459:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100232.765399:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100233.276474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100233.788561:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100234.300476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100234.812583:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100235.324477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100235.837070:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100236.350788:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100236.605591:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100236.605597:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100236.605600:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100236.605608:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4740 00000100:00080000:0.0:1766100236.605611:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100236.605616:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100236.605624:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100236.605637:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007e872964 x1851885417248576/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100236.605654:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100236.605668:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100236.605712:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417248512:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100236.605794:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766100236.605799:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417248448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100236.608573:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248576:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100236.609158:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417248512:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100236.619518:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417248448:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100236.623324:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100236.623335:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100236.623343:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007e872964 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248576:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100236.863610:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100237.372483:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100237.884716:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100238.396481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100238.908535:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100239.420767:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100239.932549:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100240.444610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100240.956573:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100241.468551:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100241.724615:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100241.724621:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100241.724625:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100241.724636:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4745 00000100:00080000:0.0:1766100241.724639:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100241.724645:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100241.724655:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100241.724670:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c1e8c233 x1851885417248768/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100241.724692:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100241.724696:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100241.724731:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417248704:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100241.724752:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248768:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100241.725633:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1766100241.725644:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417248640:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100241.728676:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1766100241.728690:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417248640:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100241.728691:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100241.728713:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417248704:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100241.728746:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c1e8c233 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248768:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766100241.981662:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100242.492531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100243.006919:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100243.517542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100244.028703:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100244.540469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100245.052944:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100245.565149:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100246.076578:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100246.588508:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100246.846084:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100246.846090:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100246.846093:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100246.846104:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4750 00000100:00080000:1.0:1766100246.846107:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100246.846143:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100246.846153:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100246.846168:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417248960/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766100246.846231:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248960:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100246.846488:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417248832:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100246.846528:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100246.846535:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417248896:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100246.847873:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100246.847893:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100246.853259:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417248896:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100246.853597:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100246.853602:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100246.853607:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417248960:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100246.853865:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417248832:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100247.100658:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100247.612593:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100248.124542:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100248.636905:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100249.149200:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100249.660515:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100250.174465:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100250.690265:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100251.198484:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100251.709065:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100251.967498:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100251.967506:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100251.967509:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100251.967519:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4755 00000100:00080000:1.0:1766100251.967522:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100251.967527:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100251.967536:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100251.967550:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d902fb2c x1851885417249152/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100251.967579:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100251.967597:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100251.968025:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249152:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100251.968456:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100251.969003:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100251.969011:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417249088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100251.988928:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417249088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100251.988946:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249024:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100251.989066:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100251.989072:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100251.989087:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249152:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100252.221451:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100252.733005:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100253.245187:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100253.756514:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100254.270145:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100254.781042:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100255.292576:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100255.806840:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100256.318445:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100256.828561:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100257.084587:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100257.084594:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100257.084598:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100257.084608:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4761 00000100:00080000:1.0:1766100257.084611:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100257.084617:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100257.084627:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100257.084641:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ca1787de x1851885417249344/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100257.084663:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100257.084666:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100257.084744:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249216:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100257.084832:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100257.084838:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417249280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100257.084928:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249344:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100257.089924:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000174f4196 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249216:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100257.089964:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d902fb2c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417249280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100257.091266:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100257.091288:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100257.091299:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ca1787de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249344:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100257.341759:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100257.857392:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100258.366009:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100258.876531:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100259.389180:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100259.900692:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100260.418154:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100260.924706:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100261.436637:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100261.948539:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100262.204620:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100262.204629:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100262.204633:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100262.204646:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4766 00000100:00080000:1.0:1766100262.204649:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100262.204658:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100262.204672:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100262.204691:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fd27a47 x1851885417249536/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1766100262.204792:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249536:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100262.204950:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249408:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100262.204990:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100262.204994:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417249472:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100262.205113:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100262.205117:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100262.210288:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000653e2b14 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249408:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100262.210512:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100262.210518:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100262.210523:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fd27a47 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249536:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100262.210656:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000403f15ff pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417249472:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100262.467127:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100262.972628:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100263.485456:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100263.996647:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100264.509131:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100265.020473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100265.533763:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100266.044535:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100266.556566:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100267.069552:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100267.324716:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100267.324725:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100267.324729:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100267.324743:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4771 00000100:00080000:3.0:1766100267.324747:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100267.324774:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100267.324788:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100267.324809:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417249728/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100267.324836:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100267.324841:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100267.324950:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417249600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100267.325080:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100267.325086:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417249664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100267.325391:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249728:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100267.330652:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417249664:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100267.330673:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417249600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100267.332307:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100267.332346:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100267.332409:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249728:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100267.581685:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100268.103351:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100268.606475:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100269.121167:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100269.629053:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100270.140544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100270.652668:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100271.164526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100271.676688:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100272.188524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766100272.444651:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417249792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100272.444828:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100272.444836:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417249856:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100272.447012:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100272.447027:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100272.447033:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100272.447048:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4776 00000100:00080000:3.0:1766100272.447051:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100272.447058:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100272.447073:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100272.447112:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417249920/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100272.447131:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100272.447134:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100272.447948:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249920:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100272.453595:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417249792:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100272.455618:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417249856:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100272.459264:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100272.459278:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100272.459290:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417249920:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100272.707995:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100273.212541:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100273.725187:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100274.239592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100274.749588:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100275.260611:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100275.776193:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100276.284462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100276.797869:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100277.308482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100277.565650:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100277.565658:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100277.565661:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100277.565672:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4781 00000100:00080000:1.0:1766100277.565675:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100277.565681:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100277.565691:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100277.565706:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f69622f7 x1851885417250112/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100277.565740:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100277.565744:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100277.566075:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249984:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100277.566172:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100277.566178:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417250048:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100277.566489:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250112:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100277.568389:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002b76164b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417250048:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100277.568409:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c23332fe pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417249984:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100277.573511:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100277.573526:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100277.573539:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f69622f7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250112:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100277.822785:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100278.332670:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100278.847709:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100279.356651:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100279.870772:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100280.380579:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100280.894262:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100281.404476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100281.917475:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100282.428475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100282.685153:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100282.685162:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100282.685165:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100282.685176:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4786 00000100:00080000:3.0:1766100282.685179:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100282.685185:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100282.685195:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100282.685210:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417250304/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100282.685240:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100282.685243:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100282.685329:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250304:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100282.687009:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250176:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100282.687085:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100282.687093:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417250240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100282.697901:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417250240:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100282.697920:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250176:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100282.703128:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100282.703156:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100282.703166:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250304:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100282.940618:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100283.452630:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100283.964599:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100284.476496:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100284.988604:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100285.500954:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100286.012550:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100286.524566:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100287.036534:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100287.549811:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100287.809569:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100287.809576:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100287.809579:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100287.809589:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4791 00000100:00080000:2.0:1766100287.809593:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100287.809599:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100287.809609:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100287.809615:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250368:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100287.809626:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000010a359ea x1851885417250496/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100287.809654:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100287.809675:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100287.809715:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100287.809724:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417250432:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100287.809893:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250496:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766100287.816275:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100287.816286:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100287.816294:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000010a359ea pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250496:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100287.817938:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000978f974b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250368:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100287.818244:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009119b13e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417250432:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766100288.060551:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100288.572485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100289.136031:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100289.596493:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100290.174562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100290.620490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100291.196536:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100291.646989:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100292.220542:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100292.668918:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766100292.924656:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417250560:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100292.932763:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100292.932772:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1766100292.932786:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:1.0:1766100292.932799:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1766100292.932803:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417250624:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100292.932809:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4796 00000100:00080000:1.0:1766100292.932813:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100292.932819:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100292.933647:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100292.933673:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007d99d644 x1851885417250688/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100292.933694:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100292.933698:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100292.933724:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250688:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100292.940833:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c9026f0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417250560:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100292.945772:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1766100292.945787:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9051765 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417250624:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100292.945790:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100292.945794:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250688:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1766100293.245120:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100293.692568:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100294.268614:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100294.716463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100295.293547:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100295.741451:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100296.316582:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100296.764552:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100297.340628:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100297.788521:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100298.045091:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100298.045100:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100298.045104:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100298.045114:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4802 00000100:00080000:3.0:1766100298.045118:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100298.045123:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100298.045133:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100298.045151:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eaa9331c x1851885417250880/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100298.045187:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100298.045190:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100298.045323:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250752:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100298.045455:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100298.045462:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417250816:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100298.045813:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250880:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1766100298.057485:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100298.057496:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100298.057507:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eaa9331c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417250880:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100298.057609:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000872f8703 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417250816:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100298.058005:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250752:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100298.364842:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100298.812482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100299.389236:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100299.836497:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100300.414441:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100300.863715:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100301.440008:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100301.884544:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100302.462110:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100302.909428:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766100303.164659:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100303.165782:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100303.165787:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100303.165791:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100303.165798:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4807 00000100:00080000:2.0:1766100303.165801:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100303.165806:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100303.165814:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100303.165828:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000db8badc0 x1851885417251072/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100303.165848:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100303.165851:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100303.165901:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251008:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100303.166007:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251072:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100303.170997:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b83cf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417250944:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100303.172125:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c471d159 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251008:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100303.175179:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100303.175199:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100303.175206:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000db8badc0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251072:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100303.484573:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100303.932476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100304.511893:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100304.957407:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100305.532578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100305.980474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100306.559901:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100307.004487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100307.583918:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100308.028539:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100308.284534:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100308.284541:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100308.284544:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100308.284554:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4812 00000100:00080000:2.0:1766100308.284557:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100308.284563:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100308.284586:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100308.284602:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002ba117bc x1851885417251264/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100308.284631:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100308.284634:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100308.284745:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251264:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100308.285794:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417251136:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100308.287253:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100308.287267:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417251200:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100308.300912:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100308.300926:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100308.300937:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002ba117bc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251264:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100308.306802:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001dc2eb3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417251136:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100308.306934:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f8448657 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417251200:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100308.605057:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100309.052471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100309.630388:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100310.076426:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100310.652678:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100311.100466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100311.677514:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100312.124463:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100312.700568:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100313.150481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766100313.405580:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417251328:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100313.409029:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100313.409036:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100313.409039:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100313.409048:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4817 00000100:00080000:1.0:1766100313.409051:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100313.409056:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100313.409066:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100313.409080:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002600e542 x1851885417251456/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100313.409103:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100313.409107:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100313.409184:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251456:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100313.409326:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417251392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100313.417257:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007d99d644 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417251328:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100313.417937:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000baf3fdb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417251392:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100313.446572:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100313.446583:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100313.446607:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251456:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100313.725057:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100314.172481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100314.748541:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100315.196458:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100315.773155:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100316.220618:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100316.797243:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100317.244468:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100317.822054:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100318.268478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100318.524566:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100318.524574:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100318.524578:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100318.524618:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4822 00000100:00080000:3.0:1766100318.524621:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100318.524626:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100318.524636:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100318.524651:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000022d7392d x1851885417251648/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100318.524673:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100318.524685:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100318.524711:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100318.524804:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100318.524810:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417251520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100318.525159:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251648:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100318.530116:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417251520:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100318.530140:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251584:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100318.538968:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100318.538978:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100318.538988:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000022d7392d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251648:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100318.847488:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100319.294139:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100319.868544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100320.318414:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100320.897437:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100321.340467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100321.916597:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100322.364460:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100322.941182:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100323.388584:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100323.644543:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100323.644550:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100323.644582:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100323.644592:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4827 00000100:00080000:3.0:1766100323.644595:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100323.644601:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100323.644613:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100323.644629:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034245f40 x1851885417251840/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100323.644647:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100323.644650:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100323.644674:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100323.644758:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100323.644764:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417251712:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100323.645268:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251840:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:02020000:1.0:1766100323.663151:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:1.0:1766100323.663164:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100323.663169:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100323.663173:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034245f40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417251840:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100323.663964:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dfb67b84 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251776:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100323.663989:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004415ca00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417251712:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100323.964510:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100324.412512:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100324.988578:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100325.436452:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100326.012593:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100326.460458:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100327.036574:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100327.486720:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100328.060561:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100328.508469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100328.764560:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100328.764568:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100328.764571:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100328.764580:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4832 00000100:00080000:3.0:1766100328.764583:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100328.764589:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100328.764600:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100328.764616:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003d231712 x1851885417252032/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1766100328.764624:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417251904:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100328.764633:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100328.764636:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100328.764662:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251968:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100328.764987:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252032:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100328.770045:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006796bf6d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417251904:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100328.776312:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417251968:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100328.780978:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100328.780990:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100328.781000:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003d231712 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252032:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100329.084732:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100329.532590:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100330.108562:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100330.559061:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100331.134511:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100331.580603:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100332.156775:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100332.608945:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100333.182022:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100333.629271:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766100333.885150:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417252096:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100333.885633:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100333.885641:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417252160:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100333.887254:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100333.887279:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100333.887283:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100333.887291:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4837 00000100:00080000:3.0:1766100333.887295:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100333.887300:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100333.887311:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100333.887334:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034ee86fc x1851885417252224/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766100333.887403:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252224:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766100333.889000:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100333.889008:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100333.898058:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417252160:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100333.898077:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417252096:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100333.898211:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100333.898228:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100333.898232:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034ee86fc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252224:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766100334.204601:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100334.653423:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100335.229951:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100335.677962:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100336.252577:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100336.700492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100337.276559:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100337.724427:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100338.300806:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100338.749562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100339.004685:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100339.004692:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100339.004696:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100339.004706:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4842 00000100:00080000:3.0:1766100339.004709:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100339.004715:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100339.004725:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100339.004757:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a64be6c0 x1851885417252416/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100339.004781:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100339.004784:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100339.004807:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417252352:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100339.004897:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100339.004902:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417252288:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100339.006783:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252416:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766100339.007959:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417252288:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100339.007976:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417252352:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100339.015077:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100339.015089:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100339.015098:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a64be6c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252416:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100339.326280:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100339.780713:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100340.348643:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100340.796542:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100341.373562:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100341.822605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100342.396935:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100342.844578:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100343.420597:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100343.868502:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766100344.124630:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417252480:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100344.130492:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100344.130500:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100344.130503:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100344.130513:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4848 00000100:00080000:1.0:1766100344.130516:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100344.130521:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100344.130531:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100344.130549:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004e20c6a4 x1851885417252608/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100344.130572:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100344.130576:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100344.130642:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252608:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100344.132018:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002600e542 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417252480:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100344.132042:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100344.132050:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417252544:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100344.135682:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100344.135693:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100344.135716:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252608:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100344.136544:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd44c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417252544:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100344.447926:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100344.892688:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100345.468576:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100345.916583:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100346.492594:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100346.940638:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100347.518044:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100347.965412:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100348.540542:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100348.988489:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100349.244552:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100349.244560:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100349.244563:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100349.244573:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4853 00000100:00080000:1.0:1766100349.244576:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766100349.244580:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417252672:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100349.244581:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100349.244591:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100349.244607:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417252800/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100349.244635:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100349.244638:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100349.244659:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417252736:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100349.244671:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252800:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1766100349.254644:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100349.254655:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100349.254665:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252800:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100349.258714:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004e20c6a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417252672:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100349.275846:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417252736:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1766100349.564781:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100350.012821:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100350.588607:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100351.039519:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100351.612596:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100352.060657:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100352.638912:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100353.084603:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100353.660560:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100354.108476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:2.0:1766100354.364627:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417252864:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100354.365893:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417252928:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100354.366900:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100354.366908:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100354.366911:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100354.366918:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4858 00000100:00080000:3.0:1766100354.366921:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100354.366927:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100354.366948:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100354.366970:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ebab33a4 x1851885417252992/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766100354.367047:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252992:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100354.378244:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006a1d607 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417252864:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100354.379077:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100354.379084:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1766100354.382106:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100354.382130:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100354.382139:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ebab33a4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417252992:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766100354.383056:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417252928:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100354.684594:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100355.132479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100355.709105:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100356.156660:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100356.737494:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100357.180469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100357.759598:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100358.208425:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100358.781132:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100359.228592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100359.485853:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100359.485861:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100359.485865:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1766100359.485870:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417253056:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100359.485871:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4863 00000100:00080000:3.0:1766100359.485875:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100359.485880:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100359.485891:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100359.485904:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a0e9007c x1851885417253184/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100359.485926:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100359.485929:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100359.485949:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417253120:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100359.486066:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253184:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766100359.490884:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417253056:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100359.491095:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000693b1cfa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417253120:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100359.491165:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100359.491170:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100359.491175:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a0e9007c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253184:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100359.807045:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100360.252478:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100360.828560:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100361.276467:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100361.855001:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100362.300857:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100362.877997:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100363.336820:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100363.900591:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100364.348481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100364.604553:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100364.604561:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100364.604564:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100364.604573:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4868 00000100:00080000:3.0:1766100364.604577:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100364.604594:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100364.604605:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100364.604620:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000005b48e82 x1851885417253376/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100364.604645:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100364.604648:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100364.604687:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253376:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100364.604727:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417253248:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100364.607866:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100364.607888:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417253312:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100364.612296:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417253248:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100364.614596:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1766100364.614605:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:1.0:1766100364.614614:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000005b48e82 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253376:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766100364.624777:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417253312:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766100364.924533:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100365.372475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100365.949529:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100366.396807:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100366.973698:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100367.421814:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100367.996577:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100368.446008:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100369.020565:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100369.468474:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100369.724598:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100369.724607:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100369.724624:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1766100369.724802:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417253440:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100369.724900:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766100369.724906:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417253504:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100369.725567:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4873 00000100:00080000:0.0:1766100369.725570:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100369.725575:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100369.725585:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100369.725603:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008fc17672 x1851885417253568/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1766100369.725649:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253568:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1766100369.727000:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100369.727006:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:3.0:1766100369.743955:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100369.743965:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100369.743973:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008fc17672 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253568:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100369.744072:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000024975221 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417253504:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100369.744082:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2c84c91 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417253440:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100370.046876:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100370.492491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100371.068629:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100371.516476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100372.097885:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100372.540646:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100373.116684:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100373.564526:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100374.142414:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100374.588586:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100374.844648:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100374.844654:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100374.844658:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100374.844667:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4878 00000100:00080000:1.0:1766100374.844670:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100374.844675:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100374.844685:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100374.844698:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000438f518e x1851885417253760/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100374.844713:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100374.844716:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100374.844736:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417253696:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100374.844818:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100374.844824:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417253632:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100374.845126:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253760:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100374.848771:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417253632:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100374.848793:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417253696:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100374.852716:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100374.852727:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100374.852736:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000438f518e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253760:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100375.165031:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100375.612479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100376.188576:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100376.636549:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100377.213960:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100377.663318:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100378.237626:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100378.684706:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100379.260584:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100379.712629:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100379.965605:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100379.965612:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100379.965615:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100379.965638:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4883 00000100:00080000:1.0:1766100379.965640:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100379.965645:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100379.965654:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100379.965666:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000094c69f2b x1851885417253952/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100379.965691:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100379.965694:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100379.965717:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417253888:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100379.965794:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100379.965799:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417253824:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100379.966692:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253952:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100379.973589:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417253888:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100379.981205:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100379.981216:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100379.981239:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417253952:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100379.981774:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417253824:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766100380.288406:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100380.734069:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100381.310475:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100381.756461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100382.334727:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100382.780479:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100383.356733:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100383.809691:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100384.380594:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100384.828545:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100385.087838:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100385.087846:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100385.087850:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100385.087860:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4889 00000100:00080000:1.0:1766100385.087864:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100385.087870:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100385.087880:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100385.087909:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417254144/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100385.087938:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100385.087941:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100385.087968:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417254080:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100385.088065:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100385.088073:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417254016:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100385.091663:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254144:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100385.095571:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417254080:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100385.095606:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417254016:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100385.105308:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100385.105323:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100385.105334:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254144:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100385.404683:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100385.852500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100386.428621:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100386.876476:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100387.454615:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100387.901824:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100388.476546:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100388.925174:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100389.501950:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100389.948561:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100390.205889:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100390.205897:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100390.205901:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100390.205912:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4894 00000100:00080000:1.0:1766100390.205914:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100390.205960:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100390.205970:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100390.205984:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f763163e x1851885417254336/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100390.206008:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100390.206012:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100390.206056:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417254272:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100390.206255:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254336:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100390.207229:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417254208:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100390.212584:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417254272:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100390.212625:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100390.212633:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100390.212639:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254336:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100390.212763:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417254208:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100390.524626:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100390.972482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100391.548553:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100391.997214:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100392.577255:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100393.020912:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100393.596571:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100394.044469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100394.623070:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100395.068605:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100395.324573:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100395.324582:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100395.324586:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100395.324597:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4899 00000100:00080000:2.0:1766100395.324601:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100395.324607:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100395.324620:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100395.324638:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009c5043f8 x1851885417254528/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100395.324653:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100395.324657:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100395.324738:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417254400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100395.324856:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100395.324863:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417254464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100395.324915:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254528:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100395.328160:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000047def236 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417254464:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100395.329026:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417254400:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100395.329116:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100395.329121:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100395.329126:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009c5043f8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254528:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100395.644582:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100396.093396:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100396.668948:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100397.116536:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100397.693131:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100398.140562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100398.718554:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100399.166382:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100399.741110:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100400.191144:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100400.444844:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100400.444852:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100400.444855:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100400.444866:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4904 00000100:00080000:3.0:1766100400.444869:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100400.444874:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100400.444884:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100400.444899:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000946e1c0a x1851885417254720/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100400.444960:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254720:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100400.445508:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417254592:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100400.445560:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100400.445567:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417254656:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100400.446595:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100400.446602:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766100400.460187:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100400.460199:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100400.460208:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000946e1c0a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254720:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1766100400.461330:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417254592:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100400.464301:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417254656:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100400.764936:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100401.212561:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100401.788569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100402.236583:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100402.812592:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100403.260538:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100403.836592:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100404.284491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100404.862525:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100405.308503:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100405.564610:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100405.564617:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100405.564620:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100405.564630:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4909 00000100:00080000:0.0:1766100405.564633:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100405.564639:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100405.564649:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100405.564664:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023592e07 x1851885417254912/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100405.564688:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100405.564705:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100405.564774:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254912:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100405.564940:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417254848:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100405.564989:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100405.564995:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417254784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100405.568241:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1766100405.568247:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d47f608b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417254784:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100405.568250:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100405.568257:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023592e07 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417254912:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100405.568275:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c52a4842 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417254848:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100405.893326:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100406.334759:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100406.908554:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100407.357470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100407.932627:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100408.384435:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100408.956544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100409.404488:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100409.980578:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100410.429940:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100410.689932:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100410.689940:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100410.689943:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100410.689953:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4914 00000100:00080000:1.0:1766100410.689956:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100410.689962:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100410.689971:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100410.689986:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002f0f1b46 x1851885417255104/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100410.690011:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100410.690027:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100410.690054:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417255040:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100410.690143:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100410.690149:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417254976:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100410.690250:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255104:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766100410.696538:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100410.696548:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100410.696558:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255104:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100410.700227:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417254976:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100410.701220:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417255040:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:1.0:1766100411.005073:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100411.452987:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100412.028681:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100412.476482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100413.057887:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100413.500527:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100414.076527:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100414.524461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100415.102551:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100415.549022:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100415.805489:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100415.805495:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100415.805499:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100415.805509:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4919 00000100:00080000:1.0:1766100415.805512:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100415.805518:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100415.805527:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100415.805540:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417255296/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100415.805565:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100415.805568:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100415.805590:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417255232:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100415.805672:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100415.805693:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417255168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100415.807154:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255296:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1766100415.812044:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100415.812054:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100415.812062:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255296:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1766100415.812305:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417255168:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100415.812321:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417255232:192.168.203.152@tcp:400:kworker/u8:2.0 00010000:00010000:0.0:1766100416.127424:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100416.572482:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100417.148757:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100417.596462:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100418.172610:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100418.620481:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100419.197305:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100419.644532:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100420.220637:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100420.668470:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100420.924660:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100420.924668:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100420.924672:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100420.924682:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4924 00000100:00080000:0.0:1766100420.924686:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1766100420.924692:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100420.924704:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100420.924720:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000292c14cb x1851885417255488/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100420.924746:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100420.924749:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100420.924801:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255488:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100420.924820:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417255360:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100420.924940:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766100420.924946:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417255424:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100420.930246:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100420.930258:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100420.930268:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000292c14cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255488:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100420.930335:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004229f3d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417255424:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100420.930345:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002326d21d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417255360:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100421.245044:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100421.695054:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100422.268550:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100422.716524:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100423.292547:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100423.740500:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100424.319115:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100424.764473:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100425.341626:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100425.788475:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100426.045635:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100426.045642:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100426.045646:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100426.045657:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4930 00000100:00080000:1.0:1766100426.045661:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1766100426.045663:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417255552:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100426.045670:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100426.045682:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100426.045702:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000002a0e366 x1851885417255680/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100426.045728:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100426.045732:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100426.045768:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417255616:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100426.046166:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255680:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100426.049439:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f7eb885 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417255552:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100426.050927:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417255616:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100426.053218:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100426.053229:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100426.053236:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000002a0e366 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255680:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766100426.365154:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100426.816886:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100427.388537:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100427.836466:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100428.412579:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100428.861922:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100429.436580:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100429.884465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100430.464544:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100430.908719:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100431.164602:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100431.164609:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100431.164613:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100431.164622:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4935 00000100:00080000:3.0:1766100431.164625:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1766100431.164630:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100431.164639:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100431.164653:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000054de8349 x1851885417255872/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100431.164671:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100431.164674:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100431.164891:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417255744:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100431.164980:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1766100431.164987:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417255808:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100431.166874:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255872:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100431.169577:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e192245 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417255808:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100431.174705:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417255744:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100431.175563:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100431.175569:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100431.175574:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417255872:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1766100431.485048:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100431.937065:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100432.509421:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100432.956533:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100433.534180:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100433.980571:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100434.557436:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100435.004592:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100435.580569:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100436.028513:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:3.0:1766100436.284630:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417255936:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100436.284887:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100436.284894:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417256000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100436.285218:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100436.285224:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100436.285240:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100436.285245:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4940 00000100:00080000:2.0:1766100436.285248:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1766100436.285253:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100436.285263:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100436.285280:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034bbf2ad x1851885417256064/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100436.285527:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1766100436.286534:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100436.286540:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1766100436.299462:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100436.299475:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100436.299485:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034bbf2ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256064:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100436.299500:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a51d758 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417256000:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100436.299518:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e377dbf8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417255936:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1766100436.607024:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100437.054204:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100437.628754:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100438.076480:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100438.662080:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100439.100594:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100439.676558:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100440.124477:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100440.700597:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100441.151881:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:0.0:1766100441.407873:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417256128:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1766100441.410942:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100441.410950:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100441.410954:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100441.410964:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4945 00000100:00080000:1.0:1766100441.410968:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1766100441.410974:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100441.410985:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100441.411001:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000454cd54f x1851885417256256/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100441.411027:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100441.411030:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1766100441.411062:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417256192:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100441.411066:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256256:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1766100441.411312:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cdd00614 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417256128:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1766100441.421982:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417256192:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1766100441.422316:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100441.422321:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100441.422325:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000454cd54f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256256:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766100441.725170:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100442.174543:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100442.748550:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100443.196471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100443.772620:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100444.220469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100444.796936:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100445.244518:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100445.821238:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100446.268461:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100446.524558:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100446.524566:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100446.524569:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100446.524580:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4950 00000100:00080000:2.0:1766100446.524584:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 6s 00000100:00080000:2.0:1766100446.524589:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100446.524600:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100446.524629:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b951c02b x1851885417256448/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1766100446.524729:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256448:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1766100446.524867:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417256320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100446.524907:0:2231:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1766100446.524914:0:2231:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417256384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1766100446.525457:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100446.525461:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100446.533628:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6bc72e5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:56772462-1282-4783-8e1a-69be7ca0dd12:2231:1851885417256384:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100446.533643:0:2231:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d30215d4-f453-4d61-9211-cb0b369f1629:2231:1851885417256320:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100446.533765:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100446.533770:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100446.533778:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256448:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766100446.845241:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100447.292871:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100447.868599:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100448.316501:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100448.892551:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100449.340491:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100449.918341:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100450.364492:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100450.941529:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100451.388581:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:2.0:1766100451.644568:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100451.644575:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1766100451.644579:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1766100451.644589:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4955 00000100:00080000:2.0:1766100451.644592:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 11s 00000100:00080000:2.0:1766100451.644599:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:2.0:1766100451.644608:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:2.0:1766100451.644658:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1f2cb20 x1851885417256640/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1766100451.644685:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1766100451.644688:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1766100451.644716:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417256576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1766100451.644801:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1766100451.644807:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417256512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100451.645218:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256640:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1766100451.646686:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000204d6b1d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417256512:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100451.650429:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b951c02b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417256576:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100451.659555:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100451.659566:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100451.659575:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1f2cb20 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256640:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100451.964714:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100452.412471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100452.988553:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100453.436805:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100454.018524:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100454.461303:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100455.036567:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100455.485921:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100456.063714:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100456.508451:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:3.0:1766100456.764569:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100456.764575:0:19981:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1766100456.764578:0:19981:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1766100456.764597:0:19981:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4960 00000100:00080000:3.0:1766100456.764602:0:19981:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 16s 00000100:00080000:3.0:1766100456.764607:0:19981:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:3.0:1766100456.764618:0:19981:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:3.0:1766100456.764633:0:19981:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000729ddac1 x1851885417256832/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1766100456.764658:0:19981:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1766100456.764661:0:19981:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100456.764686:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417256768:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1766100456.764774:0:2230:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1766100456.764780:0:2230:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417256704:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1766100456.764883:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256832:192.168.203.152@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1766100456.767910:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004951ed42 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:56772462-1282-4783-8e1a-69be7ca0dd12:2230:1851885417256768:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1766100456.769777:0:2230:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000054de8349 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d30215d4-f453-4d61-9211-cb0b369f1629:2230:1851885417256704:192.168.203.152@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1766100456.770798:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100456.770806:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100456.770813:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000729ddac1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417256832:192.168.203.152@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1766100457.084725:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100457.532469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100458.108863:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100458.556485:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100459.133229:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100459.582772:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100460.157402:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100460.604610:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100461.180774:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100461.628446:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100461.884812:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100461.884819:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100461.884823:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100461.884833:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4965 00000100:00080000:0.0:1766100461.884838:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1766100461.884841:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417256896:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100461.884843:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100461.884854:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100461.884869:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d4db789 x1851885417257024/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100461.884891:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100461.884894:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100461.884917:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257024:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100461.884939:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417256960:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100461.889209:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d602f5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417256896:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100461.889305:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1766100461.889310:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:0.0:1766100461.889314:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257024:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100461.894602:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417256960:192.168.203.152@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1766100462.206933:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100462.652471:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100463.228567:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100463.676469:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100464.252648:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100464.700494:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100465.276692:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100465.724591:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100466.302967:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100466.750257:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00100000:1.0:1766100467.004590:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417257088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100467.007367:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000968f8523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417257088:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100467.007386:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1766100467.007394:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417257152:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1766100467.007649:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100467.007654:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100467.007657:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100467.007663:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4970 00000100:00080000:0.0:1766100467.007665:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766100467.007670:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100467.007680:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100467.007701:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417257216/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100467.007730:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100467.007734:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1766100467.007752:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257216:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1766100467.010616:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d4db789 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417257152:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100467.010728:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100467.010732:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100467.010736:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257216:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1766100467.324579:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100467.772487:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100468.350906:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100468.796562:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100469.372604:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100469.820465:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100470.396581:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1766100470.844567:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100471.420584:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100471.868970:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:1.0:1766100472.124584:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100472.124591:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1766100472.124595:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1766100472.124606:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4976 00000100:00100000:0.0:1766100472.124607:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417257280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1766100472.124610:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1766100472.124617:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:1.0:1766100472.124629:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:1.0:1766100472.124647:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000016ad34f1 x1851885417257408/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1766100472.124699:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1766100472.124703:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100472.124711:0:2229:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1766100472.124720:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417257344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100472.125937:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257408:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100472.127484:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f763163e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417257344:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1766100472.127503:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002f0f1b46 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d30215d4-f453-4d61-9211-cb0b369f1629:2229:1851885417257280:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1766100472.128455:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100472.128466:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100472.128486:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000016ad34f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257408:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1766100472.448796:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100472.892534:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100473.468795:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100473.916557:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100474.492581:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100474.940437:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100475.520861:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100475.964769:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100476.540555:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1766100476.732592:0:2228:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1766100476.732615:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417257472:192.168.203.152@tcp:17:kworker/1:0.0 00000100:00100000:1.0:1766100476.735871:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000094c69f2b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:56772462-1282-4783-8e1a-69be7ca0dd12:2228:1851885417257472:192.168.203.152@tcp:17:kworker/1:0.0 00000100:00100000:3.0:1766100476.988490:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00000100:00080000:0.0:1766100477.244613:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100477.244620:0:10372:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1766100477.244623:0:10372:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1766100477.244633:0:10372:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: connect to NID 192.168.203.152@tcp last attempt 4981 00000100:00080000:0.0:1766100477.244637:0:10372:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1766100477.244642:0:10372:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff96a16011a000: import 0000000041de0fee using connection 192.168.203.152@tcp/192.168.203.152@tcp 00000008:00100000:0.0:1766100477.244652:0:10372:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 117571584, lost: 0. 00000100:00100000:0.0:1766100477.244668:0:10372:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000045031a83 x1851885417257664/t0(0) o8->lustre-OST0000-osc-ffff96a16011a000@192.168.203.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1766100477.244691:0:10372:0:(pinger.c:251:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1766100477.244695:0:10372:0:(pinger.c:265:ptlrpc_pinger_process_import()) 56772462-1282-4783-8e1a-69be7ca0dd12->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1766100477.244721:0:2229:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417257600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100477.244874:0:2228:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417257536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1766100477.244974:0:2227:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257664:192.168.203.152@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1766100477.250319:0:2229:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009cc3c5f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:56772462-1282-4783-8e1a-69be7ca0dd12:2229:1851885417257600:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1766100477.251952:0:2228:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ffe0f9f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d30215d4-f453-4d61-9211-cb0b369f1629:2228:1851885417257536:192.168.203.152@tcp:400:kworker/u8:3.0 00000100:02020000:3.0:1766100477.259259:0:2227:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff96a16011a000: operation ost_connect to node 192.168.203.152@tcp failed: rc = -19 00000100:00080000:3.0:1766100477.276717:0:2227:0:(import.c:86:import_set_state_nolock()) 0000000041de0fee lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1766100477.276731:0:2227:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.152@tcp failed (-19) 00000100:00100000:3.0:1766100477.276743:0:2227:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000045031a83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:56772462-1282-4783-8e1a-69be7ca0dd12:2227:1851885417257664:192.168.203.152@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1766100477.565152:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100478.013243:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:1.0:1766100478.588563:0:5013:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100479.036619:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:0.0:1766100479.612761:0:5012:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1766100480.060486:0:19865:0:(client.c:2500:ptlrpc_set_wait()) set 00000000e4182a15 going to sleep for 0 seconds 00010000:00010000:3.0:1766100480.648497:0:10534:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 20674 lines, 20674 kept, 0 dropped, 0 bad.